scripts.gesture_event_handlers

Authors: Carmen Meinson and Oluwaponmile Femi-Sunmaila Based on MotionInput v2 AreaOfInterest class by Ashild Kummen

class AreaOfInterest(view: scripts.tools.view.View, monitor_tracker: scripts.gesture_event_handlers.monitor_tracker.MonitorTracker, current_spacing=0)

Bases: object

Blue rectangle that maps to the corners of the screen. Autocalibrated with distance to the camera.

update_spacing_level(level: Optional[int] = None) → None

Updates the size of the AOI displayed on the view.

Parameters: level (Optional[int]) – The spacing level that should be used. if no level is given the AOI should be removed
Raises: RuntimeError – If an invalid spacing level is given (outside the range of spacing levels)

convert_xy(x_cam_percent: float, y_cam_percent: float) → Tuple[float, float]: Does an affine transformation for camera x,y coordinates to map to the screen pixels

Author: Carmen Meinson

class Smoother(smoothing_frames: int)

Bases: object

smooth(val: float) → float

class BaseMouse

Bases: object

move_cursor_pixel(pixel_x: float, pixel_y: float): moving the cursor to specified pixels on the screen. Smooth the movement of the mouse by averaging out the coordinates of the last few frames (as configured).

move_cursor_relative(change_x: int, change_y: int)

Moves the cursor relative to the current location.

Parameters

change_x (int) – change in pixels in the x direction with positive value meaning movement to the right
change_y (int) – change in pixels in the y direction with positive value meaning movement downwards

left_click()

left_press()

left_release()

right_click()

right_press()

right_release()

double_click()

scroll(speed: float)

Parameters: speed (float) – speed of scrolling, where positive speed means scrolling down and negative speed scrolling up

zoom(speed: float)

Zoom simulating the Ctrl key press with mouse scroller. Smooth the zooming by averaging out the coordinates of the last few frames (as configured).

Parameters: speed (float) – speed of the zooming, where positive speed means zooming in and negative speed zooming out

class DesktopMouse(monitor_tracker: scripts.gesture_event_handlers.monitor_tracker.MonitorTracker)

Bases: scripts.gesture_event_handlers.desktop_mouse.BaseMouse

move_cursor(screen_precent_x: float, screen_precent_y: float): Translate the screen percentages into screen pixels. Smooth the movement of the mouse by averaging out the coordinates of the last few frames (as configured). Move the mouse accordingly.

class AOIMouse(aoi: scripts.gesture_event_handlers.area_of_interest.AreaOfInterest)

Bases: scripts.gesture_event_handlers.desktop_mouse.BaseMouse

move_cursor(cam_x: float, cam_y: float): Translate the camera coordinates into the AOI coordinates. Smooth the movement of the mouse by averaging out the coordinates of the last few frames (as configured). Move the mouse accordingly.

Authors: Fawziyah Hussain and Eva Miah

class DesktopPenInput(aoi: scripts.gesture_event_handlers.area_of_interest.AreaOfInterest, Pressure=500, Erase=False)

Bases: object

move_cursor(cam_x: float, cam_y: float): Translate the camera coordinates into the AOI coordinates. Smooth the movement of the mouse by averaging out the coordinates of the last few frames (as configured). Move the mouse accordingly.

tap()

press()

release()

update_pen(): Updates the cursor position and pressure value to continue an ink stroke/drag event

update_pressure(pressure)

eraser_activate()

eraser_deactivate()

class DesktopTouch(aoi: scripts.gesture_event_handlers.area_of_interest.AreaOfInterest)

Bases: object

move_cursor(cam_x: float, cam_y: float): Translate the camera coordinates into the AOI coordinates. Smooth the movement of the mouse by averaging out the coordinates of the last few frames (as configured). Move the mouse accordingly.

singletap(): Single tap at specified coordinates and immediately release. Currently not being used.

tap(): Perform a touch down at specified coordinates

press(off_hand_coords)

Allows dragging and zooming in/out with touchpoints

Parameters: off_hand_coords (float[]) – coordinates of offhand index finger, needed for zooming

release()

class POINTER_INFO

Bases: _ctypes.Structure

ButtonChangeType: Structure/Union member

PerformanceCount: Structure/Union member

dwKeyStates: Structure/Union member

dwTime: Structure/Union member

frameId: Structure/Union member

historyCount: Structure/Union member

hwndTarget: Structure/Union member

inputData: Structure/Union member

pointerFlags: Structure/Union member

pointerId: Structure/Union member

pointerType: Structure/Union member

ptHimetricLocation: Structure/Union member

ptHimetricLocationRaw: Structure/Union member

ptPixelLocation: Structure/Union member

ptPixelLocationRaw: Structure/Union member

sourceDevice: Structure/Union member

class POINTER_TOUCH_INFO

Bases: _ctypes.Structure

orientation: Structure/Union member

pointerInfo: Structure/Union member

pressure: Structure/Union member

rcContact: Structure/Union member

rcContactRaw: Structure/Union member

touchFlags: Structure/Union member

touchMask: Structure/Union member

class Touch(FingerRadius=5)

Bases: object

initialise_contacts()

singletap(coordinates: Tuple): Peforms a touch down and then immediately releases on the specified coordinates

update(coordinates: _ctypes.Array, press: bool): Activates touchpoint at the given coordinates, or if touchpoint already active, updates coordinates to allow dragging

check_number_of_touchpoints(coordinates: _ctypes.Array) → None: Updates state depending on the number of coordinates being passed in. A touchup is performed if it is the first time multiple coordinates are in the array since the last update. This is to reset the contacts before all contacts are updated simultaneously in the next touch injection.

set_contacts(coordinate: Tuple, index: int) → None: Sets contact parameters for the specified index based on the coordinate passed in

touchup(coordinates: _ctypes.Array) → None: Releases all touchpoints on the screen

mousejiggle(coordinates: _ctypes.Array) → None: Reactivate mouse cursor since it disappears after touch is made

class EditConfigFile

Bases: object

switch_boolean(path)

Author: Jason Ho

class ExerciseActions(exercise_display: scripts.gesture_event_handlers.exercise_display.ExerciseDisplay)

Bases: object

Actions for the ExerciseEvent class. This class mainly performs keyboard actions like pressing a key, holding a key, releasing a key, as well as pressing and releasing the left and right mouse keys.

trigger_action(exercise: str, matching_count: bool, action_type: str, repeats: int = 0, key: Optional[str] = None) → None

Performs the action associated with an exercise when it is triggered.

Parameters

exercise (str) – name of exercise
action_type (str) – name of action type (key_press, key_down, left_click or right_click)
repeats (int) – number of times the exercise has been repeated
key (Optional[str]) – name of the key associated with the action, if any

held_action(action_type: str, key: Optional[str] = None) → None

Performs the action (pressing a specific key down) associated with an exercise when it is held. This is only when the action type is keydown

Parameters

action_type (str) – name of action type (key_press, key_down, left_click or right_click)
key (Optional[str]) – name of the key associated with the action, if any

release_action(action_type: str, key: Optional[str] = None) → None

Performs the action (pressing a specific key up/releasing a key) associated with an exercise when it is released.

Parameters

exercise (str) – name of exercise
action_type (str) – name of action type (key_press, key_down, left_click or right_click)
repeats (int) – number of times the exercise has been repeated
key (Optional[str]) – name of the key associated with the action, if any

clear_exercise_display() → None: Calls an exercise display method to clear the exercise text from the view

set_exercise_display(exercises: Optional[List[str]] = None) → None

Uses the config to get the exercises it needs to display in view, and calls an exercise display method to display it.

Parameters: exercises (Optional[List[str]]) – List of exercises for the count to be displayed

Author: Jason Ho

class ExerciseDisplay(view: scripts.tools.view.View)

Bases: object

Displays the name of the used exercises and the number of times they have been triggered in the view.

set_exercise_display(exercises: Optional[List[str]] = None) → None

Sets the dictionary containing the names of the exercises and starting with 0 repeats each, passing it to the DisplayElement to be displayed in the view.

Parameters: exercises (Optional[List[str]]) – List of exercises to be displayed

update_exercise_repeats(exercise: str, repeats: int) → None

Updates the dictionary with the new number of repeats for a particular exercises, passing it to the DisplayElement to be displayed in the view.

Parameters

exercise (str) – The exercise to be updated
repeats (int) – Then number of times the exercise has been repeated

clear_exericse_display() → None: Clears the displaying of the exercise text in the view by passing an empty dictionary to the DisplayElement.

Author: Jason Ho

class ExtremityActions(extremity_circles: scripts.gesture_event_handlers.extremity_circles.ExtremityCircles)

Bases: object

Actions for the ExtremityTriggerEvent class. This class mainly performs keyboard actions like pressing a key, holding a key, releasing a key, as well as pressing and releasing the left and right mouse keys.

trigger_action(extremity: str, action_type: str, repeats: int = 0, key: Optional[str] = None) → None

Performs the action (pressing a specific key down) associated with an extremity when it is triggered, if its action type is keydown. If the action type is key press or click, this does not perform it because they are done when the extremity is released. Also changes the colour of the extremity circle activated to the activated colour.

Parameters

extremity (str) – name of extremity triggered
action_type (str) – name of action type (key_press, key_down, left_click or right_click)
repeats (int) – number of times the extremity has been repeated
key (Optional[str]) – name of the key associated with the action, if any

held_action(action_type: str, key: Optional[str] = None) → None

Performs the action (pressing a specific key down) associated with an extremity when it is held. This is only when the action type is keydown.

Parameters

action_type (str) – name of action type (key_press, key_down, left_click or right_click)
key (Optional[str]) – name of the key associated with the action, if any

release_action(extremity: str, action_type: str, repeats: int = 0, key: Optional[str] = None) → None

Performs the action (pressing a specific key down) associated with an extremity when it is released. Also changes the colour of the extremity circle to the deactivated colour.

Parameters

extremity (str) – name of extremity triggered
action_type (str) – name of action type (key_press, key_down, left_click or right_click)
repeats (int) – number of times the extremity has been repeated
key (Optional[str]) – name of the key associated with the action, if any

set_extremity_circles(extremities: Optional[List[str]] = None) → None

Sets the default displaying of the extremity circles in the view, if not already displaying. This displays specified extremity triggers, or the ones that are activated if none are given.

Parameters: extremities (Optional[List[str]]) – List of extremities to display

clear_extremity_circles() → None: Clears the displaying of the extremity circles in the view.

Author: Jason Ho

class ExtremityCircles(view: scripts.tools.view.View)

Bases: object

Class for the circles representing the Extremity triggers (hit targets). These are circles displayed on the view, which when activated, change colour. They also display a number inside the circle, which represents the number of times the extremity trigger has been activated.

set_extremity_circles_dict(activated_extremities: Optional[List[str]] = None) → None

Sets the dictionary containing the names of the activated extremities and a tuple of their coordinates, activation status and repeats for the DisplayElement to be displayed in the view.

Parameters: activated_extremities (Optional[List[str]]) – List of extremities to be displayed

update_extremity_circle(extremity: str, repeats: Optional[int] = 0, activated: Optional[bool] = False) → None

Updates the colour of an extremity circle when the extremity is activated or deactivated by updating the list of extremity circles. If an extremity is activated, the circle turns to the activated colour and vice versa. Also adds the number of times the extremity has been triggered.

Parameters

extremity (str) – Name of extremity
repeats (Optional[int]) – Number of times the extremity has been triggered
activated (Optional[bool]) – If the extremity is activated

clear_extremity_circles() → None: Clears the displaying of the extremity circles in the view by passing an empty dictionary to the DisplayElement.

class Button(text: str, x: int, y: int, width: int, height: int, font_size: float, bg_colour: list[int, int, int], action: list)

Bases: object

A class used to represent a sqaure button.

Parameters

text (str) – text to display on the button
x (int) – x position of the top left corner
y (int) – y position of the top left corner
width (int) – width of the button
height (int) – height of the button
font_size (float) – size of the text in the button
bg_colour (list[int, int, int]) – background colour of the button in RBG format
action (list) – action to perform when the button is clicked, None if no special action is required

get_position() → tuple[int, int]

Gets the coordinates of the top left corner.

Returns: (x,y)
Return type: tuple[int, int]

get_size() → tuple[int, int]

Gets the width and height of the button.

Returns: (width, height)
Return type: tuple[int, int]

class KeyboardLoader

Bases: object

A class for storing and initialising all the data needed for the in air keyboard.

load_from_config() → None: Initialises the in air keyboard by getting all the data from data/config.json.

load_fron_in_air_keyboard() → None: Initialises the in air keyboard by getting all the data from data/in_air_keyboard.json.

class InAirKeyboard(view: scripts.tools.view.View, keyboard: scripts.gesture_event_handlers.keyboard.Keyboard, kita_transcriber: scripts.gesture_event_handlers.transcription.Transcriber)

Bases: object

A class used to represent an in air keyboard.

Parameters

view (View) – the window that the in air keyboard is to be displayed on
keyboard (Keyboard) – instance of keyboard object used for pressing keys and performing key combinations
kita_transcriber (Transcriber) – instance of Ask Kita Transcriber object used for transcription

initialise_hands() → None: Initliases the data strucuture required for validating key presses from click gestures.

update_buttons(layout: list[str]) → None

Creates a button for each key in layout.

Parameters: layout – keys you want to create buttons for

create_button(key: str) → None

update_button_coordinates(button: scripts.gesture_event_handlers.in_air_keyboard.Button) → None

get_attributes(key_details: dict) → list

Gets the attributes required for creating a button.

Parameters: key_details (dict) – details of the key stored in data/in_air_keyboard.json
Returns: [width, height, font_size, bg_colour]
Return type: list

toggle_caps_lock() → None

press_special_key(key: str) → None

open_link(link: str) → None

switch_keyboard_layout(layout: list[str]) → None

speech_to_text() → None

click(hand: str, gesture: str, time_held: float) → None

release(hand: str, gesture: str) → None

key_selection(landmarks: dict) → None

Checks if the specified landmarks are currently hovering over a key, or if a key is being clicked, every frame, and passes the set of hovered and clicked keys to the view to update the display accordingly.

Parameters: landmarks (dict) – a dictionary storing the required landmark coordinates for multitouch

check_clicks(hand: str, gesture: str, button: scripts.gesture_event_handlers.in_air_keyboard.Button, pressed_keys: set, keys_to_click: set) → None: Uses time held value for a given gesture to determine whether to click keys, and to differentiate between doing a single press and key hold.

get_closest_key(landmark: tuple[float, float]) → tuple[tuple, scripts.gesture_event_handlers.in_air_keyboard.Button]

For a given landmark, finds the closest button in the current layout. It first finds the closest x-coordinate to the landmark from all buttons in the current layout and then finds the closest y-coordinate from the buttons with just that x-coordinate.

Parameters: landmark (tuple[float, float]) – a tuple of floats storing x,y coordinates for the given landmark
Return landmark_position, button: returns landmark position relative to camera window and the closest button
Return type: tuple[float, float], Button

find_closest_value(num: float, lst: list) → float: Searches through the pre-sorted input list of floats to find the point of insertion for the input number, using bisect_left, and then returns whichever value is closest to the input by comparing against adjacent values.

check_if_inside_button_boundary(position: tuple[float, float], button: scripts.gesture_event_handlers.in_air_keyboard.Button) → bool

click_buttons(buttons: list[scripts.gesture_event_handlers.in_air_keyboard.Button]) → None

get_key_details(button: scripts.gesture_event_handlers.in_air_keyboard.Button) → dict

clear_keyboard_keys() → None

class Keyboard

Bases: object

up_arrow_press()

down_arrow_press()

right_arrow_press()

left_arrow_press()

key_press(key)

key_down(key)

key_up(key)

cut()

copy()

paste()

undo()

print()

switch()

help()

close()

windows_key()

windows_run()

macros()

right_arrow()

left_arrow()

escape()

full_screen()

space()

page_up()

page_down()

volume_up()

volume_down()

save()

press(key): added in this function as self.key_press doesn’t support pressing the full range of the keys required in the InAirKeyboard event handler e.g. print screen, page down etc.

key_combination(keys)

press_release_keys(keys, press)

type_write(phrase)

Author: Oluwaponmile Femi-Sunmaila

class MonitorTracker

Bases: object

get_monitor_size() → Tuple[int, int]

Returns: monitor size in pixels [width, height]
Return type: Tuple[int,int]

get_screen_ratio() → float

Returns: screen ration defined as width/height
Return type: float

change_monitor() → None: Changes the active monitor aka the monitor in relation to which the size and the xy coordinates are calculated.

convert_xy(x_screen_percent: float, y_screen_percent: float) → Tuple[float, float]: Convert the percentage coordinates of the screen to the pixel coordinates of the current monitor in relation to the main one

Author: Andrzej Szablewski

class NoseBox(view: scripts.tools.view.View, monitor_tracker: scripts.gesture_event_handlers.monitor_tracker.MonitorTracker)

Bases: object

update_nose_box(nose_point: Optional[Tuple[float, float]] = None) → None

remove_nose_box()

update_nose_box_centre(nose_box_centre: Tuple[float, float]) → None

Authors: Fawziyah Hussain and Eva Miah Partially based on MotionInput v2 Touch code by Chenuka Ratwatte

class POINTER_INFO

Bases: _ctypes.Structure

ButtonChangeType: Structure/Union member

PerformanceCount: Structure/Union member

dwKeyStates: Structure/Union member

dwTime: Structure/Union member

frameId: Structure/Union member

historyCount: Structure/Union member

hwndTarget: Structure/Union member

inputData: Structure/Union member

pointerFlags: Structure/Union member

pointerId: Structure/Union member

pointerType: Structure/Union member

ptHimetricLocation: Structure/Union member

ptHimetricLocationRaw: Structure/Union member

ptPixelLocation: Structure/Union member

ptPixelLocationRaw: Structure/Union member

sourceDevice: Structure/Union member

class POINTER_TOUCH_INFO

Bases: _ctypes.Structure

orientation: Structure/Union member

pointerInfo: Structure/Union member

pressure: Structure/Union member

rcContact: Structure/Union member

rcContactRaw: Structure/Union member

touchFlags: Structure/Union member

touchMask: Structure/Union member

class POINTER_PEN_INFO

Bases: _ctypes.Structure

penFlags: Structure/Union member

penMask: Structure/Union member

pointerInfo: Structure/Union member

pressure: Structure/Union member

rotation: Structure/Union member

tiltX: Structure/Union member

tiltY: Structure/Union member

class PEN_TOUCH_UNION

Bases: _ctypes.Union

penInfo: Structure/Union member

touchInfo: Structure/Union member

class POINTER_TYPE_INFO

Bases: _ctypes.Structure

DUMMYUNIONNAME: Structure/Union member

type: Structure/Union member

class Pen

Bases: object

pentap(coordinates: _ctypes.Array, pressure, erase): Single pen tap at given coordinates

pendown(coordinates: _ctypes.Array, pressure, erase): Initialises a pen down event for ink strokes/drag events

penup(coordinates: _ctypes.Array): Lifts pen up after taps or drags

update_pen_info(coordinates: _ctypes.Array, pressure): Sets updated coordinates and pressure for ink strokes

mousejiggle(coordinates: _ctypes.Array): The mouse cursor dissapears after pen down event, so we shall move the mouse a little to reactivate it

Author: Jason Ho

class Keyboard

Bases: object

key_press(key: str) → None

Performs a keypress as a DirectX action.

Parameters: key (str) – name of key to press

key_down(key: str) → None

Performs a keydown as a DirectX action.

Parameters: key (str) – name of key for keydown

key_up(key: str) → None

Performs a keyup as a DirectX action.

Parameters: key (str) – name of key for keyuo

class Clicker

Bases: object

left_click() → None: Performs a left click as a DirectX action.

right_click() → None: Performs a left click as a DirectX action.

class Transcriber

Bases: object

start_transcribe()

stop_transcribe()

Module contents

Authors: Carmen Meinson

class GestureEventHandlers(mode_change_method: Callable, view)

Bases: object

get_handler_func(handler_name: str, function_name: str)