gamemechanics.progamemechanics.pro / Mechanics
ENRU
GitHub
← All mechanic references
Mechanic map

Phone camera interactions

Games, learning, creativity and interactive applications. Author's systematization of game rules based on proven technical capabilities. Examples are application ideas and not statements of released products.

Families: 21Variants: 168Updated: 2026-10-20
Read MarkdownJSONSuggest an edit

Mechanics catalog

1Framing and visual attentionOverview and example ↗What the player shows to the camera and how he places it in the frame. The position in the image is not the same as the position in the room. To automatically check, you need tracking, a marker or a limited scene.8

Variants and applications

  • Targeting. Action: Combine an object with the sight to select or activate it. Application: Photo hunting, exhibit selection, evidence scanner. Technologies: Image analysis; Recognition or machine learning model.
  • Staying in frame. Action: Do not let the target leave the specified area until the scale is filled. Application: Taming a creature, long-term scanning, tracking. Technologies: Image analysis; Recognition or machine learning model.
  • Specified scale. Action: Achieve the required portion of the frame area by moving the phone closer or changing the zoom. Application: Preparing the image, managing the charge, selecting the size. Technologies: Image analysis.
  • Composition according to a template. Action: Place several elements in the marked areas. Application: Photography training, advertising shooting, composition puzzles. Technologies: Image analysis; Recognition or machine learning model.
  • Combination of contours. Action: Apply a real contour to a translucent sample. Application: Search for a shooting point, scene reconstruction, silhouette assembly. Technologies: Image analysis.
  • Prospective combination. Action: Find the angle in which spaced objects form a figure. Application: Anamorphic riddles, illusions of scale, visual passwords. Technologies: Image analysis.
  • Frame composition. Action: Show the desired combination of objects and exclude prohibited ones. Application: Orders for a photographer, visual recipes, sorting objects. Technologies: Recognition or machine learning model.
  • Scanning the sector. Action: Inspect parts of the scene one by one; the game takes into account coverage. Application: Search for a missing person, room inspection, virtual panorama. Technologies: Image analysis; AR spatial tracking.
Context and sources: Apple AVFoundation: Camera Capture and Control, MediaPipe Object Detector
2Capture timing and recording modesOverview and example ↗An event is the receipt of an image or a segment of video. The picture can be checked by an algorithm or used as a result of creativity. Capture frequency and latency vary by device.8

Variants and applications

  • A photograph as an action. Action: Press the shutter to freeze the state and make a game move. Application: Photo attack, collecting evidence, trading card. Technologies: Camera capture.
  • Decisive moment. Action: Take a frame while the target is in a short acceptable state. Application: Reporting, photo hunting, rhythmic shooting. Technologies: Camera capture; Recognition or machine learning model.
  • Serial shooting. Action: Take several shots and choose a successful one within the limit. Application: Catching a quick event, comparison of takes, editorial game. Technologies: Camera capture.
  • Observation excerpt. Action: Record continuously the specified time; the gap resets progress. Application: Behavioral research, detective surveillance, video task. Technologies: Camera capture; Recognition or machine learning model.
  • Delayed self-timer. Action: Start the timer and have time to assemble a composition or take a pose. Application: Group tasks, cooperative, staged photography. Technologies: Camera capture.
  • Event shutter. Action: The camera automatically captures an object's input, gesture or change. Application: Camera trap, automation, cause-and-effect experiments. Technologies: Camera capture; Recognition or machine learning model.
  • Stop-motion animation. Action: Move real objects between separate pictures. Application: Stop-motion, bringing toys to life, teaching animation. Technologies: Camera capture.
  • Changing the time scale. Action: Record the process and play it back quickly or slowly. Application: Observation of transformations, trick analysis, editing puzzle. Technologies: Camera capture; Device capabilities.
Context and sources: Apple AVFoundation: Camera Capture and Control
3Optics and camera settingsOverview and example ↗The player controls how the image is obtained. Digital effects can be simulated in software. Optical zoom, bulb exposure, focus and lens selection are not available with every camera and API combination.8

Variants and applications

  • Change zoom. Action: Zoom in or out to find a detail or select a target. Application: Virtual binoculars, investigation, training shooting. Technologies: Camera capture.
  • Transfer of focus. Action: Switch sharpness between near and far objects. Application: Selecting a layer of the puzzle, learning to focus, deciphering details. Technologies: Camera capture; Device capabilities.
  • Focus as a parameter. Action: Rotate the knob to maximum sharpness in the selected area. Application: Lens simulator, device setup, macroquest. Technologies: Camera capture; Device capabilities.
  • Exposition. Action: Change the brightness of the capture and look for the working range of the image. Application: Teaching photography, restoring visible details, setting up the scanner. Technologies: Camera capture; Device capabilities.
  • Exposure and lubrication. Action: Select the exposure duration according to the movement. Application: Action freezing, artistic trace, photographic simulator. Technologies: Camera capture; Device capabilities.
  • White balance. Action: Match the color temperature of the image with the standard. Application: Color calibration, photo school, art assignments. Technologies: Camera capture; Device capabilities.
  • Lens selection. Action: Switch between available viewing angles and focusing distances. Application: Macro search, wide view, perspective tasks. Technologies: Camera capture; Device capabilities.
  • Flash or flashlight. Action: Turn on the backlight to change the visible scene or run a game rule. Application: Study of a dark layout, development of reflections, scanner mode. Technologies: Camera capture; Device capabilities.
Context and sources: Apple AVFoundation: Camera Capture and Control, Android CameraX: zoom, torch, focus, exposure, Android: multi-camera and hardware limitations
4Light, color and shadowsOverview and example ↗Simple visual quantities are extracted from the image. Auto exposure and auto white balance change the measurements. An uncalibrated camera is not a replacement for a light meter or colorimeter.8

Variants and applications

  • Brightness as a regulator. Action: Bring a light or dark surface to the camera to change the parameter. Application: Instrument volume, creature energy, setting virtual light. Technologies: Image analysis.
  • Light threshold. Action: Move between lit and dark scene to switch state. Application: Day and night, hidden world, light puzzle. Technologies: Image analysis.
  • Showing color. Action: Bring a colored object; its shade is determined by a command or resource. Application: Mixing paints, color spells, teaching colors. Technologies: Image analysis.
  • Collection of palette. Action: Find several desired shades in your surroundings. Application: Collecting, creative walks, design selection. Technologies: Image analysis.
  • Color sequence. Action: Show colors in a given order and tempo. Application: Visual code, music sequencer, memory game. Technologies: Image analysis.
  • Light drawing. Action: Move a small light source; the program accumulates its trajectory. Application: Drawing in the air, runes, creating light creatures. Technologies: Image analysis.
  • Shadow as a controller. Action: Move or change a shadow on a prepared surface. Application: Shadow theatre, catching targets, shape puzzles. Technologies: Image analysis.
  • Lighting direction. Action: Rearrange the light source and evaluate shadows or highlights in a controlled scene. Application: Educational laboratory, portrait setup, optical puzzles. Technologies: Image analysis; Own algorithm.
Context and sources: Apple AVFoundation: Camera Capture and Control, Android CameraX: zoom, torch, focus, exposure
5Physical manipulation of the imageOverview and example ↗The player does not change the scene in the game, but the path of light to the lens. Blacking out does not prove finger touching. For mirrors, filters and refraction, prepared scenes with clear conditions are more convenient.8

Variants and applications

  • Close the lens. Action: Completely block the image to hide, switch the world or pause. Application: Horror, hide and seek, tactile command. Technologies: Image analysis.
  • Partially block the frame. Action: Open and close image areas; the game takes into account the share or position of the overlap. Application: Curtain, picture development, visual controller. Technologies: Image analysis.
  • The rhythm of the ceilings. Action: Alternate closed and open states according to a temporary pattern. Application: Code lock, rhythm game, visual alphabet. Technologies: Image analysis.
  • Color filter. Action: Look through a safe transparent color film and change the available color channel. Application: Optical experiments, prepared ciphers, changing the world mode. Technologies: Image analysis; Prepared props.
  • Mirror view. Action: Look at the target through the mirror and control the reflected image. Application: Riddles with reflections, periscope, unusual angle. Technologies: Image analysis; Prepared props.
  • Refractive medium. Action: Look through a transparent object and use image deformation. Application: Optical puzzles, artistic effects, virtual lens. Technologies: Image analysis; Prepared props.
  • Physical aperture. Action: Look through a hole or mask and combine visible fragments. Application: Spyglass, searching for a secret, image restoration. Technologies: Image analysis; Prepared props.
  • Screen recursion. Action: Point the camera at the screen with its own image and use a repeating pattern. Application: Video art, recursive labyrinths, visual synthesizer. Technologies: Image analysis; Prepared props.
Context and sources: Apple AVFoundation: Camera Capture and Control
6Motion in the image planeOverview and example ↗The flow of frames turns into directions, speeds and trajectories. Without additional cues, camera movement and object movement are easy to confuse. Need a fixed phone, markers or motion compensation.8

Variants and applications

  • Entrance to the area. Action: Bring a hand or object into the marked area to press a virtual button. Application: Non-contact interface, children's game, reaction. Technologies: Image analysis.
  • Crossing the line. Action: Move an object through the virtual boundary in the desired direction. Application: Goal, pass counter, conveyor sorting. Technologies: Image analysis.
  • Movement speed. Action: Move the target faster or slower, changing the force of the action. Application: Impact, character acceleration, visual musical instrument. Technologies: Image analysis.
  • Flow direction. Action: Move the scene or controller left, right, up or down. Application: Steering, swiping, touchless navigation. Technologies: Image analysis.
  • Trajectory as a symbol. Action: Draw a circle, zigzag or other shape with the target's movement. Application: Runes, gesture passwords, flight drawing. Technologies: Image analysis.
  • Fading. Action: Keep the image or selected object still. Application: “Freeze”, sight stabilization, virtual neutralization. Technologies: Image analysis.
  • Repeat route. Action: Draw the tracked point along the pattern with an acceptable deviation. Application: Coordination training, labyrinth, copying a picture. Technologies: Image analysis.
  • Area of ​​change. Action: Move so as to fill the required portion of the frame with changes. Application: Acceleration of the elements, collective activity, abstract dance. Technologies: Image analysis.
Context and sources: Apple AVFoundation: Camera Capture and Control, MediaPipe Object Detector
7Objects, signs and relationshipsOverview and example ↗Real things become input to the rules. The finished detector recognizes its classes. Recognizing a specific toy, material or condition usually requires a separate model and testing on real scenes.8

Variants and applications

  • Item category. Action: Show an object of the desired class; the game accepts it as an answer. Application: Scavenger hunt, dictionary, resource search. Technologies: Recognition or machine learning model.
  • Specific instance. Action: Show a pre-registered item or its visual label. Application: Personal artifact, level key, collectible toy. Technologies: Recognition or machine learning model; Own algorithm.
  • Counting objects. Action: Display a specified number of visible items of one or more classes. Application: Arithmetic, order picking, visual inventory. Technologies: Recognition or machine learning model.
  • Geometric shape. Action: Find a circle, triangle, symmetrical or elongated object. Application: Teaching geometry, collecting parts, visual classification. Technologies: Image analysis.
  • Material or texture. Action: Show a surface from a limited trained set. Application: Texture catalog, artistic search, conditional crafting resources. Technologies: Recognition or machine learning model; Own algorithm.
  • Visual condition. Action: Translate a familiar object between distinguishable states. Application: Open box as a switch, model assembly, checking a task step. Technologies: Recognition or machine learning model; Own algorithm.
  • Spatial relationship in the frame. Action: Place one object above, inside a contour or next to another. Application: Visual recipes, teaching prepositions, compositional riddles. Technologies: Image analysis; Recognition or machine learning model.
  • Comparison with a sample. Action: Find a thing or image similar to the standard according to the selected criterion. Application: Matching a pair, searching for differences, visual memory. Technologies: Image analysis; Recognition or machine learning model.
Context and sources: MediaPipe Object Detector, ARCore Augmented Images
8Text, pictures and machine codesOverview and example ↗The camera translates visible signs into discrete commands. OCR depends on the writing and the quality of the frame. ML Kit Text Recognition v2 lists Latin, Chinese, Devanagari, Japanese, Korean; For the Cyrillic alphabet, a different approach with proven support is needed.8

Variants and applications

  • The printed word. Action: Read a word from a card, sign or page and apply it as a command. Application: Language game, word hunt, password search. Technologies: Text recognition.
  • Number or expression. Action: Read numbers in the prepared format and check the condition. Application: Arithmetic, digital lock, tasks with real price tags. Technologies: Text recognition.
  • Handwriting on paper. Action: Write a symbol or word; camera recognizes limited alphabet. Application: Teaching writing, paper console, answering without touching the screen. Technologies: Recognition or machine learning model; Own algorithm.
  • QR as a trigger. Action: Scan a code containing the identifier of an item, task or session. Application: Quests, museum routes, starting a level. Technologies: Code recognition.
  • Product barcode. Action: Turn the packaging code into a key for a local table or generator. Application: Collecting, household roguelike, sorting. Technologies: Code recognition.
  • Fiducial marker. Action: Show geometry label; its ID, angle and size set the parameters. Application: Physical controller, figurine positioning, laboratory. Technologies: Markers.
  • Image key. Action: Point the camera at a pre-prepared poster, page or illustration. Application: A book that comes to life, an advertising quest, a museum layer. Technologies: Markers; AR spatial tracking.
  • Optical sequence. Action: Read changing frames or light states from another screen. Application: Game code transmission, device connection, visual protocol. Technologies: Image analysis; Own algorithm.
Context and sources: ML Kit Text Recognition v2: supported scripts, ML Kit Barcode Scanning, ARCore Augmented Images
9Paper, cards and tabletop objectsOverview and example ↗The player changes the physical layout, and the phone reads its state. Dice, manuscript and unmarked tiles require separate processing. Convenient starting configuration: fixed camera, contrast field, unique markers.8

Variants and applications

  • Digitization of the drawing. Action: Draw a silhouette on paper and transfer it to the game scene. Application: Own character, children's creativity, paper editor. Technologies: Image analysis.
  • Drawn level. Action: Draw walls, platforms or paths with conventional signs. Application: Maze editor, teaching algorithms, physical level design. Technologies: Image analysis.
  • Paper panel. Action: Touch or cover the drawn areas under the camera. Application: Homemade remote control, music keyboard, interactive poster. Technologies: Image analysis.
  • Layout of cards. Action: Change the set and order of recognized cards in order to change the program or recipe. Application: Programming for children, writing a story, crafting. Technologies: Markers.
  • Physical chips. Action: Rearrange and rotate tracked pieces on the field. Application: Digital board game, tactical combat, logistics simulation. Technologies: Markers.
  • Reading dice. Action: Roll contrasting dice; the camera reads the upper edges. Application: Tabletop automation, probabilistic experiments, party game. Technologies: Image analysis; Own algorithm.
  • Constructor under the camera. Action: Assemble a figure from distinguishable parts and run its digital verification. Application: Geometric tasks, digital twin of a building, training. Technologies: Recognition or machine learning model; Own algorithm.
  • Paper warp. Action: Fold or rotate marked paper, changing visible areas and their connections. Application: Origami puzzle, folding controller, paper interface. Technologies: Markers; Own algorithm.
Context and sources: ML Kit Barcode Scanning, ARCore Augmented Images
10Face and head positionOverview and example ↗The front camera becomes a pose and presence controller. Facial landmarks do not confirm identity. Face size approximation; turning your head does not give you a point of view on the screen.8

Variants and applications

  • Presence of a person. Action: Appear in front of the camera or exit the frame to enable the game state. Application: Hide and seek, personal scene entry, interactive mirror. Technologies: Recognition or machine learning model.
  • Head position. Action: Move your head in the frame and control the character or cursor. Application: Contactless arkanoid, evasion, accessibility interface. Technologies: Recognition or machine learning model.
  • Turn to the sides. Action: Turn your head left and right to select direction. Application: Steering, response selection, view control. Technologies: Recognition or machine learning model.
  • Head tilt. Action: Tilt your head to your shoulder and change a continuous parameter. Application: Balance, flight, tilt of a virtual surface. Technologies: Recognition or machine learning model.
  • Nod or denial. Action: Perform a sequence of tilts or turns for a discrete command. Application: Dialogue, confirmation, rhythm game. Technologies: Recognition or machine learning model.
  • Approaching face. Action: Change the visible scale of the face relative to the calibration. Application: Inflating a virtual bubble, zoom, comic transformation. Technologies: Recognition or machine learning model.
  • Maintaining head posture. Action: Stay in the desired angle range for a given time. Application: Balancing game, beam holding, precise targeting. Technologies: Recognition or machine learning model.
  • Several faces. Action: Gather the required number of participants in the frame and evaluate their location. Application: Group portrait quest, cooperative gate, party game. Technologies: Recognition or machine learning model.
Context and sources: MediaPipe Face Landmarker
11Eyes, mouth and facial actionsOverview and example ↗The game rule reacts to the visible action of a person. Need calibration and alternate input. The facial form does not prove emotion; blinking is involuntary. Accurate eye tracking with an RGB camera should not be promised.8

Variants and applications

  • Blinking like a button. Action: Intentionally close and open your eyes to perform an action. Application: Shooting, jumping, experimental contactless input. Technologies: Recognition or machine learning model.
  • Wink. Action: Close left and right eyes separately for different commands. Application: Two-button game, hidden signals, side control. Technologies: Recognition or machine learning model.
  • Closed eyes. Action: Keep your eyes closed; the world changes over time or when opened. Application: Horror, memory game, audio-visual mystery. Technologies: Recognition or machine learning model.
  • Rough direction of view. Action: After calibration, distinguish several large directions of eye movement. Application: Choice of two zones, experiment on attention, character control. Technologies: Recognition or machine learning model; Experimental prototype.
  • Opening the mouth. Action: Change the degree of mouth opening as a continuous parameter. Application: Eating targets, vocal avatar without sound analysis, virtual instrument. Technologies: Recognition or machine learning model.
  • Smile and eyebrows. Action: Take a given mouth shape or raise eyebrows for a command. Application: Facial trainer, comic arcade, avatar control. Technologies: Recognition or machine learning model.
  • Repeat expression. Action: Reproduce the combination of facial movements shown. Application: Facial memory, acting, accuracy competition. Technologies: Recognition or machine learning model.
  • Mimic animation. Action: Transfer facial parameters to a digital character in real time. Application: Virtual theater, animation creation, video chat character. Technologies: Recognition or machine learning model.
Context and sources: MediaPipe Face Landmarker
12Hands, fingers and hand gesturesOverview and example ↗Hands give positional, discrete and continuous commands. You need to take into account mirroring, overlapping hands, and falling out of frame. The position of the fingers does not mean that the system understands any sign language.8

Variants and applications

  • Finger pointing. Action: Move your fingertip and hover the virtual pointer. Application: Touchless menu, drawing, target selection. Technologies: Recognition or machine learning model.
  • Pinch. Action: Bring your thumb and index finger together to grasp, unclench to release. Application: Drag and drop, assembly, working with virtual objects. Technologies: Recognition or machine learning model.
  • Fist and palm. Action: Squeeze and unclench the hand to change the binary state. Application: Shield and attack, holding an object, launching an effect. Technologies: Recognition or machine learning model.
  • Number of fingers. Action: Show a number with your fingers in a consistent set of gestures. Application: Arithmetic, answer choice, numeric code. Technologies: Recognition or machine learning model.
  • Turn the palm. Action: Change the angle of the hand to rotate a handle, shield or object. Application: Spatial puzzle, device control, gestural music. Technologies: Recognition or machine learning model.
  • Two-handed zooming. Action: Spread, bring together and rotate two brushes. Application: Sizing and rotating an object, stretching a field, sculpting. Technologies: Recognition or machine learning model.
  • Gesture trajectory. Action: Run a sequence of movements with your palm or finger. Application: Spells, dance game for hands, robot commands. Technologies: Recognition or machine learning model.
  • Manual pantomime. Action: Reproduce a gesture from a limited trained vocabulary. Application: Rock, paper, scissors, charades, learning selected gestures. Technologies: Recognition or machine learning model; Own algorithm.
Context and sources: MediaPipe Hand Landmarker
13Posture, body movement and active playOverview and example ↗The phone is installed so that the camera sees the necessary parts of the body. The player needs a visible and free area of ​​space. Posture assessments are approximate and do not replace diagnostics or professional assessment of exercise technique.8

Variants and applications

  • The body is like a joystick. Action: Move the body relative to the calibrated center. Application: Runway control, flight, balancing arcade. Technologies: Recognition or machine learning model.
  • Repeat the pose. Action: Match the position of the main joints with the sample. Application: “Wall of poses”, dance movement training, living sculpture. Technologies: Recognition or machine learning model.
  • Squatting and standing up. Action: Cross calibrated pelvic and knee position thresholds. Application: Evasion, jump charge, motor mini-game. Technologies: Recognition or machine learning model.
  • Raising hands and steps. Action: Perform visible actions with the desired limb. Application: Rhythm game, warm-up, coordination tasks. Technologies: Recognition or machine learning model.
  • Angles and amplitude. Action: Achieve a given joint position range. Application: Motor puzzle, avatar size control, coordination training. Technologies: Recognition or machine learning model.
  • Counting repetitions. Action: Perform several completed pose cycles in the correct order. Application: Active tasks, combo series, motor counter. Technologies: Recognition or machine learning model.
  • Reproduction of movement. Action: Repeat a temporal sequence of poses taking into account the tempo. Application: Dancing, choreography, character imitation. Technologies: Recognition or machine learning model.
  • Retention and balance. Action: Maintain a pose within an acceptable range for a specified time. Application: “Freeze”, balancing game, world stability control. Technologies: Recognition or machine learning model.
Context and sources: MediaPipe Pose Landmarker
14Silhouette, masks and segmentationOverview and example ↗The game works with the shape of the occupied area, not just the joints. The quality of the mask depends on the model, light, clothing and background. Collision in 2D does not mean physical contact in 3D.8

Variants and applications

  • Silhouette as a collider. Action: Cross virtual targets with a body contour or hit particles. Application: Chamber Pong, catching balls, interactive installation. Technologies: Recognition or machine learning model.
  • Passage through the hole. Action: Fit the silhouette to the virtual opening. Application: “Wall with a hole”, spatial coordination, group puzzle. Technologies: Recognition or machine learning model.
  • Occupying zones. Action: Close the necessary parts of the frame with your body and leave others open. Application: Live buttons, command pattern, dynamic poster. Technologies: Recognition or machine learning model.
  • Body drawing. Action: Leave a trace of a mask or individual parts of the body. Application: Video art, motion painting, creative play. Technologies: Recognition or machine learning model.
  • Silhouette as a surface. Action: Apply a material, pattern or game state to a selected person. Application: Digital costume, coloring, character disguise. Technologies: Recognition or machine learning model.
  • Virtual layer overlap. Action: Hide a digital object behind a selected person in a 2D composition. Application: Focus, animated mirror, interactive scene. Technologies: Recognition or machine learning model.
  • Group form. Action: Create a common silhouette from several participants. Application: Cooperative sculpture, letters with bodies, party game. Technologies: Recognition or machine learning model.
  • Changing the environment. Action: Save a person and replace the background or its individual classes. Application: Virtual stage, video theater, creative portrait. Technologies: Recognition or machine learning model.
Context and sources: MediaPipe Pose Landmarker, MediaPipe Image Segmenter
15AR: surfaces and spatial geometryOverview and example ↗The camera, together with tracking, connects digital objects with the surrounding space. Conventional AR often combines video and IMU. Depth does not necessarily require LiDAR, but support and accuracy vary by device and scene. Glass and mirrors are difficult to reconstruct.8

Variants and applications

  • Placement on the surface. Action: Find a table, floor or wall and attach a digital object to it. Application: Tabletop strategy, furniture arrangement, miniature world. Technologies: AR spatial tracking.
  • Link to image. Action: Attach a digital object to a recognized page, map or poster. Application: A book that comes to life, a card game, interactive instructions. Technologies: Markers; AR spatial tracking.
  • Measuring space. Action: Select points and estimate the distance, area or suitable size. Application: Geometric tasks, scaling of buildings, model fitting. Technologies: AR spatial tracking.
  • Real obstacles. Action: Use estimated furniture and wall geometry for digital collisions. Applications: Balls around the room, arena, terrain puzzles. Technologies: AR spatial tracking; Sensor or depth estimation.
  • Deep overlap. Action: Hide virtual objects behind real surfaces using a depth map. Application: Hide and seek, search for creatures, convincing AR scene. Technologies: AR spatial tracking; Sensor or depth estimation.
  • Filling the volume. Action: Use a spatial map for virtual water, snow or fill. Application: Flooding simulation, terraforming, creative effects. Technologies: AR spatial tracking; Sensor or depth estimation.
  • The surface is like a route. Action: Build a path or game grid using recognized geometry. Application: Indoor tower defense, robot route, spatial puzzle. Technologies: AR spatial tracking; Own algorithm.
  • Geometry as a condition. Action: Find a large enough free surface or a suitable configuration. Application: Selecting an arena, finding a location for a base, educational spatial puzzle. Technologies: AR spatial tracking; Own algorithm.
Context and sources: ARCore: motion tracking, planes, anchors, ARCore Depth: depth and occlusion, Apple ARKit: world tracking
16AR: viewpoint and observation rulesOverview and example ↗The player acts by moving the phone and changing what he sees. Screen boundaries, line of sight in the model, and human attention are different quantities. Returning to a place requires relocalization, and consistency between sessions does not occur automatically.8

Variants and applications

  • The phone is like a window. Action: Move the device to look into a fixed digital world. Application: Diorama, virtual aquarium, object exploration. Technologies: AR spatial tracking.
  • Walking around the object. Action: Examine a digital or recognized object from different angles. Application: Finding a weak point, museum model, spatial riddle. Technologies: AR spatial tracking.
  • The angle is like a key. Action: Find the point where the elements of a digital scene add up to answer. Application: Perspective puzzle, character alignment, hidden caption. Technologies: AR spatial tracking.
  • Distance as action. Action: Enter the range of distances relative to the virtual object. Application: Taming, activating a portal, playing with caution. Technologies: AR spatial tracking.
  • Observation holds the state. Action: Keep the object in the visible sector; when you turn away, it changes. Application: Horror, “keep in sight”, quantum puzzle. Technologies: AR spatial tracking.
  • Action outside the frame. Action: Remove an object from view and return it to trigger the transition. Application: Tricks, transformations, observation puzzle. Technologies: AR spatial tracking.
  • Portal and border crossing. Action: Guide the phone or player through a given virtual area. Application: Another room, changing the world scale, spatial quest. Technologies: AR spatial tracking.
  • Return to anchor. Action: Re-find a previously saved location and continue the state of the object. Application: Permanent pet, abandoned cache, asynchronous scene. Technologies: AR spatial tracking; Device capabilities.
Context and sources: ARCore: motion tracking, planes, anchors, ARCore Depth: depth and occlusion, Apple ARKit: world tracking
17Filming as game content creationOverview and example ↗The result of the capture becomes the material for the next game action. The transformation can be a simple algorithm, an editor, or a model. For generative methods, latency, cost, and robustness of the result need to be checked separately.8

Variants and applications

  • Photo texture. Action: Capture a surface and apply it to a digital object. Application: Clothing creation, base design, material library. Technologies: Camera capture.
  • Photo like a sprite. Action: Cut an object out of the frame and use it as a character or object. Application: Playing with your own toys, collage, custom inventory. Technologies: Camera capture; Recognition or machine learning model.
  • Portrait as an avatar. Action: Create a character's appearance based on a selected photo. Application: Personalization, story comics, group characters. Technologies: Camera capture; Recognition or machine learning model.
  • The stage is like a level. Action: Convert image contours or spots into game geometry. Application: Procedural maze, picture platformer, photo puzzle. Technologies: Image analysis.
  • Image as seed. Action: Stably convert image attributes into generator parameters. Application: Packaging monster, daily dungeon, visual crafting. Technologies: Image analysis.
  • Visual music. Action: Match colors, lines or positions of objects to notes and timbres. Application: Chamber sequencer, music teaching, sound design. Technologies: Image analysis.
  • Three-dimensional copy. Action: Take an object from several angles and reconstruct the model. Application: Digitization of toys, museum objects, asset creation. Technologies: Scene reconstruction; Device capabilities.
  • Meaning generation. Action: Use the description of the recognized scene to create a task or story. Application: Quest from room objects, story from a photo, AI game master. Technologies: Recognition or machine learning model; Generative model.
Context and sources: Apple AVFoundation: Camera Capture and Control, MediaPipe Image Segmenter, MediaPipe Object Detector
18Time, video memory and state comparisonOverview and example ↗The game links several observations of one scene. For comparison, alignment and taking into account light changes are needed. Photos and recording times alone do not prove the fact, location, or authorship of an event.8

Variants and applications

  • Find change. Action: Compare the scene with a standard and detect an added, removed or moved object. Application: Detective, memory, assembly control. Technologies: Image analysis; Recognition or machine learning model.
  • Restore the scene. Action: Reproduce the previous layout using a saved image. Application: Puzzle cleaning, evidence reconstruction, spatial memory. Technologies: Image analysis; Recognition or machine learning model.
  • Combination with the past. Action: Overlay the previous frame and combine the angle or position of the object. Application: Repeating a photograph, animation, observing changes. Technologies: Image analysis.
  • Video echo. Action: Show delayed recording and interact with your past movement. Application: Duet with yourself, time puzzle, video performance. Technologies: Camera capture.
  • Phantom trajectory. Action: Save a path or pose and repeat or bypass it. Application: Coordination, asynchronous competition, movement pattern training. Technologies: Image analysis; Recognition or machine learning model.
  • Looped video part. Action: Record a short action and build it into a repeating game loop. Application: Video music, live-action characters, editing game. Technologies: Camera capture.
  • Sequence of actions. Action: Check not only the finale, but also the order of the visible steps. Application: Recipe, assembly, procedure training, visual combo. Technologies: Recognition or machine learning model; Own algorithm.
  • Changes over a long time. Action: Capture one scene at intervals and turn observed changes into progress. Application: Building diary, creative project, observing plant growth. Technologies: Camera capture; Recognition or machine learning model.
Context and sources: Apple AVFoundation: Camera Capture and Control
19Multiple cameras, participants and video callingOverview and example ↗Camera interaction is distributed between people or filming points. Simultaneous capture by different cameras depends on the model and mode. A collaborative AR scene requires synchronization of coordinates and game state; The video stream itself does not provide it.8

Variants and applications

  • Front/rear switching. Action: Change the camera aimed at the player and at the environment as two game modes. Application: “Me and the world”, change of role, transfer of action between the sides of the phone. Technologies: Camera capture.
  • Front and rear at the same time. Action: Link the facial action with what is happening in front of the rear camera. Application: Mimic spell on an object, reaction and event, double scene. Technologies: Camera capture; Device capabilities.
  • Two rear angles. Action: Use available cameras with different fields of view simultaneously. Application: Review and sighting, part search, experimental stereo scene. Technologies: Camera capture; Device capabilities.
  • Common field for several phones. Action: Combine local maps and show one spatial game. Application: AR tabletop, joint construction, local battle. Technologies: AR spatial tracking; Network exchange.
  • Mirror duet. Action: Repeat the visible movements of another participant in real time. Application: Dance duel, synchronization, team coordination. Technologies: Recognition or machine learning model; Network exchange.
  • Remote observer. Action: One player shows the world with a camera, the other guides it via video link. Application: Cooperative detective, quest operator, remote guide. Technologies: Camera capture; Network exchange.
  • Separate information. Action: Show different participants different cameras or digital layers. Application: Asymmetric cooperative, ciphers, search with an operator. Technologies: Camera capture; Network exchange.
  • Visual relay race. Action: Pass a frame, gesture or layout to the next participant to continue. Application: Chain story, photo quest, group animation. Technologies: Camera capture; Network exchange.
Context and sources: Apple AVFoundation: Camera Capture and Control, ARCore: motion tracking, planes, anchors, Android: multi-camera and hardware limitations
20Real world search and visual tasksOverview and example ↗Chamber actions are included in full game cycles. These are the component mechanics: search plus capture, recognition and progression. Assignments must take into account the available environment; recognition allows for error and human confirmation.8

Variants and applications

  • Photo hunt according to the list. Action: Consistently remove suitable objects and close the task points. Application: City quest, vocabulary training, children's expedition. Technologies: Recognition or machine learning model.
  • Visual bestiary. Action: Open collection records based on images of a permitted set of objects. Application: Toy catalog, nature study, collectible game. Technologies: Recognition or machine learning model; Own algorithm.
  • Evidence with conditions. Action: Fix the desired object with the desired detail or in the desired state. Application: Detective, reporter quest, investigative training. Technologies: Recognition or machine learning model; Own algorithm.
  • Order a photo. Action: Perform a combination of theme, composition, moment and constraints. Application: Photographer simulator, creative training, competitive tasks. Technologies: Camera capture.
  • Environment as resources. Action: Use visual attributes of found things to obtain game materials. Application: Urban crafting, walking game, resource puzzle. Technologies: Recognition or machine learning model.
  • Real action changes the world. Action: Change the physical scene, after which the digital system updates the state. Application: Sorting on a table, building a shelter for a virtual creature, puzzle. Technologies: Recognition or machine learning model; Own algorithm.
  • Route according to visual landmarks. Action: Find a sequence of familiar images or prepared objects. Application: Museum quest, navigation riddle, thematic excursion. Technologies: Markers; Recognition or machine learning model.
  • Mission from the local scene. Action: Suggest a task from what the system actually detected around. Application: Adaptive scavenger hunt, everyday learning, game without fixed props. Technologies: Recognition or machine learning model; Own algorithm.
Context and sources: MediaPipe Object Detector, ML Kit Text Recognition v2: supported scripts, ARCore Augmented Images
21Hybrid and experimental methodsOverview and example ↗The camera participates in the system along with other inputs or a complex model. This is where it is especially important to test the prototype on target phones. These options are not universal capabilities of any camera.8

Variants and applications

  • Camera and microphone. Action: Coordinate visible action with sound: gesture with word, movement with clap. Application: Multimodal commands, rhythm game, stage performance. Technologies: Combination of channels; Sound.
  • Camera and IMU. Action: Match the video with the rotation and acceleration of the device. Application: Spatial controller, sustainable AR tracking, phone movement as a tool. Technologies: Combination of channels; AR spatial tracking.
  • Camera and location. Action: Combine a visual landmark with GPS and, if available, visual positioning. Application: Urban AR game, content at a landmark, route. Technologies: Combination of channels; Geolocation.
  • Camera and touch screen. Action: Select an area by touch, then associate it with a real object or effect. Application: Interactive cutting, tracking a selected target, drawing on top of the world. Technologies: Combination of channels; Image analysis.
  • Light exchange between devices. Action: The screen or flashlight of one device sends a code to the camera of another. Application: Local connection, game key, physical riddle channel. Technologies: Combination of channels; Own algorithm.
  • Visual management agent. Action: The model interprets the limited scene and suggests the next move or reaction. Application: Adaptive presenter, procedural learning, experimental NPC. Technologies: Recognition or machine learning model; Generative model; Experimental prototype.
  • Recognition of compound action. Action: To associate an object, gesture and scene change by time. Application: Analysis of a visual recipe, interactive story, complex combo. Technologies: Recognition or machine learning model; Own algorithm.
  • Performance style. Action: Compare the rhythm, smoothness or expressiveness of visible movement with a selected criterion. Application: Creative challenge, evaluation of an animated sketch, dance game. Technologies: Recognition or machine learning model; Own algorithm; Experimental prototype.
Context and sources: Apple AVFoundation: Camera Capture and Control, ARCore: motion tracking, planes, anchors, ARCore Geospatial, W3C Media Capture and Streams

Borders

Mechanics = human action → observed signal → system rule → result or feedback.

The 168 entries cover the major interaction families. The list is not mathematically exhaustive: combinations, new models and new equipment create additional mechanics. Some of the recordings are composite game loops.

How to use the catalog

Separate player action, signal recognition, game rule, and feedback. The related variants here preserve the context of application and are not necessarily independent physical primitives.

When prototyping, test one rule and failure conditions first. The presence of an API, sensor, or source does not prove that all scenarios work. For devices, test feature availability, calibration, and alternate input on the actual target hardware.

Sources and examples

Sources support particular techniques or methodology, not every family variant. Design examples and mechanic combinations are illustrative.

CC BY 4.0 · gamemechanics.pro contributors