scripts.gesture_events

Author: Carmen Meinson

class ClickPressEvent(gesture_type: str, additional_bodypart_types={})

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Clicking/Pressing with the gesture_type specified. Currently as there is no logic of how to decide if a press or a click should be started press is trigerred only if the click trigger was not given

[trigger types]:: “click”: called once when the gesture of interest becomes active “press”: called when the gesture of interest becomes active IF the “click” trigger type was not defined “release”: called when the gesture of interest becomes unactive IF the “click” trigger type was not defined
[bodypart types]:: “hand”: the hand on which we track the gesure of interest
[gestures types]:: defined by the gesture_type arg.

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

force_deactivate() → None: As some events need to finish some processing before being deactivated, they can implement this method. It is called when the events are removed from the model. (e.g. for example if an event has called the mouse press function it needs to release the mouse before deactivation)

class SingleHandExclusiveClickPressEvent(gesture_type: str)

Bases: scripts.gesture_events.click_press_event.ClickPressEvent

Clicking/Pressing only if the same gesture is NOT made on the off_hand. RN used so that mouse would not be clicked during zooming as both use index_pinch

Author: Jason Ho

class ExerciseEvent(exercise_name: str, mode: str)

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Class representing an exercise event. Each event is an exercise e.g. walking, and the gestures it handles correspond to the exercise states that it has activated e.g. walking_left_state. Allows for actions including key press, holding a key down, and mouse clicks.

[trigger types]:: “set_exercise_display”: called in the event set_up function to display the chosen exercises on the view “triggered”: called when an exercise state that wasn’t activated before has been activated “held”: called when an exercise state that was already activated remains activated “released”: called when an exercise state that was already activated is now deactivated “clear”: called in the event force_deactivate function to remove the exercise state from the view
[bodypart types]:: “body”: using gestures that are activated by body landmarks
[gestures types]:: defined by the activate exercise states of the activated exercise

get_all_used_gestures() → set

Returns the set of the gestures that the exercise event depends on.

Returns: set of gestures that the exercise depends on
Return type: set

get_exercise_name() → str

Returns name of the exercise that the event represents.

Returns: name of exercise
Return type: str

update() → None: If the state of the exercise is True, checks the state of the gestures that the exercise depends on and triggers gesture event handler (exercise actions) methods (triggered, held, released) if conditions are met. These actions are either a key press, a key down (holding the key), and left or right mouse clicks.

set_up() → None: Sets the displaying of the activated exercises in the View

force_deactivate() → None: Force deactivates the exercise by ‘releasing’ all gesture types/exercise states of the exercise, and calling the gesture event handler (exercise actions)’s clear method, which is used to remove the exercise text from the view.

Author: Jason Ho

class ExtremityTriggerEvent(extremity_name: str)

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Class representing an extremity trigger event. Each event corresponds to an extremity trigger (represented as a circle in the view). Allows for actions including key press, holding a key down, and mouse clicks, and changing the colour of the extremity circle.

[trigger types]:: “set_exercise_display”: called in the event set_up function to display the default extremity circles on the view “triggered”: called when an extremity trigger that wasn’t activated before has been activated “held”: called when an extremity trigger that was already activated remains activated “released”: called when an extremity trigger that was already activated is now deactivated “clear”: called in the event force_deactivate function to remove the extremity circles from the view
[bodypart types]:: “body”: using gestures that are activated by body landmarks
[gestures types]:: defined by the gesture corresponding to the represented extremity trigger

get_all_used_gestures() → set

Returns the names of the associated gestures with the extremity, which is normally the extremity names.

Return gesture_types: names of the associated gesture (e.g. “{arm_left}” or “{up}”)
Rtype gesture_types: set

update() → None: If the state of the extremity trigger is True, checks the state of the gestures that the extremity trigger depends on and triggers gesture event handler (extremity actions) methods (triggered, held, released) if conditions are met. These actions are either a key press, a key down (holding the key), and left or right mouse clicks, as well as colouring the corresponding extremity circle to represent whether it is activated or deactivated.

set_up() → None: Sets the displaying of the extremity circles in the View

force_deactivate() → None: Force deactivates the exercise by deactivating all gesture types/extremity triggers of the exericise, and calling the gesture event handler (extremity actions)’s clear method, which is used to remove the extremity circles from the view.

Author: Jason Ho

class ExtremityWalkingEvent(gesture_types: Optional[set] = {'arm_left_extremity', 'arm_right_extremity', 'up_extremity', 'walking_left_state', 'walking_right_state'}, event_trigger_types: Optional[set] = {'clear', 'extremity_released', 'extremity_triggered', 'held', 'set_displays', 'walking_released', 'walking_triggered'})

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Class representing the walking + extremity triggers mode. Allows for walking on the spot to trigger a key holding down action, with 3 extremity triggers to change the key that is being held down.

[trigger types]:: “set_displays”: called in the event set_up function to display the extremity circles and walking exercise in the view “walking_triggered”: called when a walking state that wasn’t activated before has been activated “extremity_triggered”: called when an extremity trigger that wasn’t activated before has been activated “held”: called when a walking state that was already activated remains activated (not extremity trigger) “walking_released”: called when a walking state that was already activated is now deactivated “extremity_released”: called when an extremity trigger that was already activated is now deactivated “clear”: called in the event force_deactivate function to remove the extremity circles and walking exercise display from the view
[bodypart types]:: “body”: using gestures that are activated by body landmarks
[gestures types]:: “walking_left_state”: state for walking (lifting left leg) with the left leg “walking_right_state”: state for walking (lifting right leg) with the right leg “arm_left_extremity”: left extremity trigger to change the current key bind “arm_right_extremity”: right extremity trigger to change the current key bind “up_extremity”: head extremity trigger to change the current key bind

update() → None: If the state of the exercise is True, checks the state of the gestures that the exercise depends on and triggers gesture event handler (exercise actions) methods (triggered, held, released) if conditions are met. These actions are either a key down if the gesture is a walking state, or colouring an extremity circle.

set_up() → None: Sets the displaying of the extremity circles and activated exercises in the View

force_deactivate() → None: Force deactivates the exercise by ‘releasing’ all gesture types/exercise states , and calling the gesture event handler (exercise actions)’s clear method, which is used to remove the exercise text and extremity circles from the view.

class GamingExtremityWalkingEvent

Bases: scripts.gesture_events.extremity_walking_event.ExtremityWalkingEvent

Class representing the walking + extremity triggers gamepad mode. Special mode to allow for walking to perform an action along with holding an extremity trigger, as well as standalone extremity triggers to act as buttons.

[trigger types]:: “set_displays”: called in the event set_up function to display the extremity circles and walking exercise in the view “walking_triggered”: called when a walking state that wasn’t activated before has been activated “extremity_triggered”: called when an extremity trigger that wasn’t activated before has been activated “held”: called when a walking state or a button extremity trigger that was already activated remains activated “walking_released”: called when a walking state that was already activated is now deactivated “extremity_released”: called when an extremity trigger that was already activated is now deactivated “clear”: called in the event force_deactivate function to remove the extremity circles and walking exercise display from the view
[bodypart types]:: “body”: using gestures that are activated by body landmarks
[gestures types]:: “walking_left_state”: state for walking (lifting left leg) with the left leg “walking_right_state”: state for walking (lifting right leg) with the right leg “left_walking_extremity”: left walking extremity trigger to activate with walking to cause a key down action “right_walking_extremity”: right walking extremity trigger to activate with walking to cause a key down action “up_walking_extremity”: up walking extremity trigger to activate with walking to cause a key down action “down_walking_extremity”: down walking extremity trigger to activate with walking to cause a key down action “down_button_extremity”: down button extremity trigger to cause a key down action “right_button_extremity”: right button extremity trigger to cause a key down action “up_button_extremity”: up button extremity trigger to cause a key down action “left_button_extremity”: left button extremity trigger to cause a key down action “start_key_extremity”: start key extremity trigger to cause a key down action “quit_key_extremity”: quit key extremity trigger to cause a key down action

update() → None: If the state of the exercise is True, checks the state of the gestures that the exercise depends on and triggers gesture event handler (exercise actions) methods (triggered, held, released) if conditions are met. These actions are either a key down if the gesture is a walking state, or colouring an extremity circle.

class EyeMode2Event

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

font = 0

duration = 500

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

move_cursor(x, y, ymax, xmax, ymin, xmin)

class EyeTrackingEvent

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

_summary_

[trigger types]: [bodypart types]: [gestures types]:

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

Author: Carmen Meinson

class GesturesActiveEvent(desired_gesture_states: Dict[str, Dict[str, bool]], frames_nr=1)

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Calls the ‘active’ trigger once all the specified gestures have been in the specified states for the given nr of frames

[trigger types]:: “active”: called when all the specified gestures have been in the specified states for the given nr of frames
[bodypart types]:: retrieved from the desired_gesture_states
[gestures types]:: retrieved from the desired_gesture_states

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

Author: Carmen Meinson

class HandActiveEvent

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Each frame the facing_camera gesture is active, the ‘move’ trigger is called with the coordinates of the palm center

[trigger types]:: “move”: called every frame when the event is active (so when the hand is facing the camera) with input of (palm_center_x_coordinate, palm_center_y_coordinate). coordinates in percentage of the camera view relative to the left top corner
[bodypart types]:: “hand”: which hands movement is tracked
[gestures types]:: “facing_camera”:

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

Author: Carmen Meinson

class IdleStateChangeEvent(currently_idle=False)

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Switching between the idle(hand resting or not facing the camera) and active hand states.

[trigger types]:: “idle”: called when currently_idle is False and the state of the hand has been idle for frames_for_switch frames “active”: called when currently_idle is True and the state of the hand has been active for frames_for_switch frames
[bodypart types]:: “hand”: the hand on which we track the gesture of interest
[gestures types]:: “facing_camera”: the hand is considered idle when the gesture is inactive

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

class KeyboardActiveEvent

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Every frame this event is active, the “move” trigger is called, passing in the landmarks specified in config, for each active hand in the frame. A hand is considered “active” when it is facing the camera, i.e. when the gesture “facing_camera” for that hand is available.

[trigger types]:: “move”: called every frame as long as this event is active “clear”: called in the event force_deactivate function to remove the in air keyboard display from the view
[bodypart types]:: “dom_hand”: the dominant hand being tracked “off_hand”: the off hand being tracked
[gestures types]:: “facing_camera”: active when the specified hand is facing the camera

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

get_landmarks()

force_deactivate() → None: As some events need to finish some processing before being deactivated, they can implement this method. It is called when the events are removed from the model. (e.g. for example if an event has called the mouse press function it needs to release the mouse before deactivation)

class KeyboardClickEvent

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

This event is active when any one of the gestures of interest is currently active. Click event trigger passes the hand, gesture name and the time that gesture has been held for, when called. Release event trigger passes the hand and the name of the gesture that is no longer active when called.

[trigger types]:: “click”: called when any one of the gestures of interest becomes active “release”: called when a gesture of interest which was previously active becomes inactive
[bodypart types]:: “dom_hand”: the dominant hand on which we track the gestures of interest “off_hand”: the off hand on which we track the gestures of interest
[gestures types]:: defined in config under the keyboard event settings.

initialise_gestures() → None

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

Author: Radu-Bogdan Priboi

class MouthTriggerEvent(gesture_types)

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Left Clicking/Pressing when mouth is opened. Currently as there is no logic of how to decide if a press or a click should be started. Right Clicking is triggers by smiling and Double Clicking by raising eyebrows.

[trigger types]:

“click”: called once when the gesture of interest becomes active; “press”: called when the gesture of interest becomes

active IF the “click” trigger type was not defined;

“release”: called when the gesture of interest becomes unactive

[bodypart types]:

“head”

[gestures types]:

“smiling_gesture” & “open_mouth_gesture” & “raise_eye_brow_gesture”

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

class SmilingEvent(key=None): Bases: scripts.gesture_events.mouth_trigger_event.MouthTriggerEvent

class OpenMouthEvent(key=None): Bases: scripts.gesture_events.mouth_trigger_event.MouthTriggerEvent

class RaiseEyeBrowEvent(key=None): Bases: scripts.gesture_events.mouth_trigger_event.MouthTriggerEvent

class FishFaceEvent(key=None): Bases: scripts.gesture_events.mouth_trigger_event.MouthTriggerEvent

class MoveNoseBoxEvent(phrase: str)

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

update() → None: Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

Author: Andrzej Szablewski

class NoseDirectionTrackingEventNoseBox

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Nose Direction tracking event with visual nosebox and nosepoint. Because of the visual features, always active when face and the nosepoint can be detected (nose_position_gesture). Mouse movement active only when at least one of other nose gestures is active. Defines the mouse direction vector from the center of the nosebox and nosepoint. Using the vector and the scaling_factor relatively moves mouse in that direction.

[trigger types]:

“move”: relatively moves mouse in the direction of the vector from the: center of the nosebox to the nosepoint.

“nose_box”: visualizing nosebox and current nosepoint position

[bodypart types]:

“head”

[gestures types]:

“nose_right_gesture”: active when the nosepoint is to the right of: the nosebox.
“nose_left_gesture”: active when the nosepoint is to the left of: the nosebox.

“nose_up_gesture”: active when the nosepoint is above the nosebox. “nose_down_gesture”: active when the nosepoint is below the nosebox. “nose_position_gesture”: active when a face and the nosepoint can be

detected.

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

force_deactivate() → None: remove nose box when mode was changed

Author: Andrzej Szablewski

class NoseDirectionTrackingEvent

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

CURRENTLY NOT IN USE, more optimial version of the NoseDirectionTrackingEventNoseBox without visuals! Nose Direction tracking event. Active if face can be detected and at least 1 gesture is active i.e. nose leaves the nosebox (defined by the noseboxpercentagesize). Defines the mouse direction vector from the center of the nosebox and the nosepoint. Using the vector and the scaling_factor relatively moves mouse in that direction.

[trigger types]:

“move”: relatively moves mouse in the direction of the vector from the: center of the nosebox to the nosepoint.

[bodypart types]:

“head”

[gestures types]:

“nose_right_gesture”: active when the nosepoint is to the right of: the nosebox.
“nose_left_gesture”: active when the nosepoint is to the left of: the nosebox.

“nose_up_gesture”: active when the nosepoint is above the nosebox. “nose_down_gesture”: active when the nosepoint is below the nosebox.

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

Author: Radu-Bogdan Priboi

class NoseScrollEvent

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Scrolling by smiling and moving the nose up (to scroll up) or down (to scroll down). Scrolling speed is proportional to the distance from the tip of the nose to the centre.

[trigger types]:: “scroll”: called every frame after frames_for_switch as long as the speed is above threshold. With arguments (speed) reflecting the distance from the nose tip to the centre.
[bodypart types]:: “head”
[gestures types]:: “nose_scroll_up_gesture” & “nose_scroll_down_gesture”: gestures activated when smiling is detected and nose tip is up/down

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

Author: Andrzej Szablewski

class NoseTrackingEvent

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Nose tracking event. Active if face can be detected. Maps nosepoint position to the screen coordinate and moves mouse cursor in that direction.

[trigger types]:: “move”: maps normalized position to screen coordinates and moves mouse cursor there.
[bodypart types]:: “head”
[gestures types]:: “nose_position_gesture”: gesture always active when face is detected and nosepoint position is available

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

force_deactivate() → None: As some events need to finish some processing before being deactivated, they can implement this method. It is called when the events are removed from the model. (e.g. for example if an event has called the mouse press function it needs to release the mouse before deactivation)

Author: Radu-Bogdan Priboi

class NoseZoomEvent

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Zooming by doing a fishface and moving the nose to left (to zoom in) or right (to zoom out). Zooming speed is proportional to the distance from the tip of the nose to the centre.

[trigger types]:

“zoom”: called every frame after frames_for_switch as long as the speed is above threshold. With arguments (speed) reflecting the distance from the nose tip to the centre.

[bodypart types]:

“head”

[gestures types]:

“nose_zoom_in_gesture” & “nose_zoom_out_gesture”: gestures activated: when fishface is detected and nose tip to left/right

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

Author: Carmen Meinson

class PalmHeightChangeEvent(active_level=None)

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Palm height level is an array where each element represents the lower bound of the range so if the possible ranges for the levels were [0, 12), [12, 23), [23, inf) the array would be [0,12,23] in the legacy code [0, 0.18-0.075, 0.18-0.25] lower the level the further the hand is from the camera

[trigger types]:: “level_change”: called when the palm height has been in a size bracket from different from active_level for frames_for_switch frames
[bodypart types]:: “hand”: the hand on which we track the gesure of interest
[gestures types]:: “facing_camera”: the palm height levels change only when the hand is facing the camera.

set_up() → None: sets the aoi level to either active_level that was given as an arg on initialization or to the level 0 in order for it to be displayed

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

force_deactivate() → None: As some events need to finish some processing before being deactivated, they can implement this method. It is called when the events are removed from the model. (e.g. for example if an event has called the mouse press function it needs to release the mouse before deactivation)

Author: Carmen Meinson

class ScrollEvent

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Scrolling by holding up middle and index finger. Scrolling speed and direction determined by the distance on y axis between the tip and the upper joint of the middle finger.

[trigger types]:: “scroll”: called every frame after frames_for_switch as long as the change in distance since the last frame is above threshold and the distance between the middle and index finger tips are below the threshold. With arguments (speed) reflecting the change in distance since last frame.
[bodypart types]:: “hand”: the hand to scroll with
[gestures types]:: “scroll”: the current gesure used

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

Author: Carmen Meinson

class SimpleGestureEvent(gesture_types: Set[str], event_trigger_types: Set[str], bodypart_types: Set[str])

Bases: scripts.core.gesture_event.GestureEvent

Parent class to be used by standard GestureEvents. Provides the methods that guarantee that all the gestures of interest are in the self._gestures after the notify methods are called by the model.

set_trigger_functions(funcs: Dict[str, Callable]) → None

set_bodypart_names(name_to_type: Dict[str, str])

get_state() → bool

notify_gesture_activated(gesture: scripts.core.gesture.Gesture)

Notifies that a new gesture of the same type that is used by the GestureEvent has activated. If the activated gesture has the correct body part name and gesture type combination, it is stored in the GestureEvent instance. The state of the GestureEvent is then checked in case the activation of the gesture changed it.

Parameters: gesture (Gesture) – the gesture that activated

notify_gesture_deactivated(gesture: scripts.core.gesture.Gesture)

Notifies that a new gesture of the same type that is used by the GestureEvent has deactivated. If the deactivated gesture was previously stored in the GestureEvent instance it is now removed (although there may be cases where it is kept to perform calculations later). The state of the GestureEvent is then checked in case the deactivation of the gesture changed it.

Parameters: gesture (Gesture) – the gesture that deactivated

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

Authors: Samuel Emilolorun

class SpeechEvent(phrase: str)

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

_summary_

[trigger types]: [bodypart types]: [gestures types]:

update() → None: Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

class TouchPressEvent(gesture_type: str)

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Pressing using touchpoints. Holding the gesture of interest with just the dominant hand activates 1 touchpoint. Holding with both hands activates 2 touchpoints.

[trigger types]:: “click”: called when the gesture of interest first becomes active “press”: called when the gesture of interest is already active (gesture being held) “release”: called when the gesture of interest becomes inactive
[bodypart types]:: “dom_hand”: the dominant hand on which we track the gestures of interest “off_hand”: the off hand on which we track the gestures of interest
[gestures types]:: defined by the gesture_type arg.

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

force_deactivate() → None: As some events need to finish some processing before being deactivated, they can implement this method. It is called when the events are removed from the model. (e.g. for example if an event has called the mouse press function it needs to release the mouse before deactivation)

Author: Carmen Meinson

class ZoomEvent

Bases: scripts.gesture_events.simple_gesture_event.SimpleGestureEvent

Zooming by holding the index pinch gesture on both hands. Zooming direction and speed according tho the change in the index finger tips distance.

[trigger types]:: “zoom”: called every frame after frames_for_switch as long as the change in distance since the last frame is above threshold. With arguments (speed) reflecting the change in distance since last frame.
[bodypart types]:: “dom_hand” & “off_hand”: the distance (aka speed) is calculated from the distance between the “index_tip”s of these hands
[gestures types]:: “index_pinch”: the current gesture used

update(): Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

Module contents

Author: Carmen Meinson

class GestureEvents

Bases: object

static get_event(name: str)