scripts.core

Author: Carmen Meinson

class Model

Bases: object

add_module(module_name: str, module: scripts.core.module.Module) None
get_module_names() Set[str]
Returns

names of all modules that are currently active in the model

Return type

Set[str]

switch_events(events_to_remove: Set[str], events_to_add: Dict[str, scripts.core.gesture_event.GestureEvent]) None

First removes the specified events from the model and adds new ones in afterwards. Removing an event may also lead to removing of some gestures that are no longer used by any events in the model. If all gestures from one module get removed so does the module.

Parameters
  • events_to_remove (Set[str]) – names of the events to remove

  • events_to_add (Dict[str, GestureEvent]) – names of the events to add mapped to the event instances

add_gesture(module_name: str, gesture_name: str, primitives: Set[scripts.core.gesture_factory.Primitive]) None

Add a gesture to the given module. If the gesture is already in the module then nothing is done. (this means that if there is an attempt to add a new gesture by the same name as a previous one, it will be ignored)

Parameters
  • module_name (str) – module to which to add the gesture to

  • gesture_name (str) – name of the gesture to be added

  • primitives (Set[Primitive]) – primitives that describe the gesture

Raises

RuntimeError – if the module we are trying to add the gesture to is not in the model

remove_gesture(gesture_name: str) None

Simply removes the gesture from the model so that no new instances of this gesture will be created. Any currently active instances of this gesture will not be deactivated (are expected to deactivate naturally) Does not check if any events were using it so removing a gesture in use may lead to an event that cant be activated

Parameters

gesture_name (str) – name of the gesture to remove

Raises

RuntimeError – if a gesture with the given name has not been added to the model

process_frame(frame: numpy.ndarray) scripts.core.raw_data.RawData
Process the frame in following steps:
  • Process the frame in each module, from which we obtain the frames raw data and new Gesture instances may be created.

  • Activate in the model each Gesture that was created from the modules, which may (de)activate of some gesture events

  • Update all the currently active gestures, which may result in deactivation of some of them, and consequently (de)activation of some gesture events

  • Run all active gesture events, which may result in the trigger functions in the events being called.

Parameters

frame (ndarray) – image reflecting the frame

Returns

Coordinates of all the landmarks detected in the frame

Return type

RawData

Author: Carmen Meinson

class LandmarkDetector

Bases: object

Interface for each modules landmark detector

get_raw_data(raw_data: scripts.core.raw_data.RawData, image: numpy.ndarray) None

Adds the xy(z) coordinates of all the landmarks detected on the image into the RawData instance.

Parameters
  • raw_data (RawData) – RawData instance to add the landmarks to

  • image (ndarray) – Image to process with mediapipe and read the landmarks locations from

class Module

Bases: object

update_and_get_activated_gestures(frame_data: scripts.core.raw_data.RawData, image: numpy.ndarray) Set[scripts.core.gesture.Gesture]
  • Use the modules landmark detector (ML library) to retrieve the RawData (aka the coordinates of all landmarks) from the frame.

  • Update all position trackers with the RawData, which results in a set of primitives that had changed since the last frame.

  • Update all the gesture factories that use the changed primitives, which may create new active Gestures.

Parameters
  • frame_data (RawData) – RawData instance to store the detected landmarks in

  • image (ndarray) – frame

Returns

set of the new Gestures that were created

Return type

Set[Gesture]

add_gesture(gesture_name: str, primitives: Set[scripts.core.gesture_factory.Primitive]) None

Adds a new available gesture to the module.

Parameters
  • gesture_name (str) – name of the gesture to be added

  • primitives (Set[Primitive]) – primitives that describe the gesture

remove_gesture(gesture_name: str) None

Remove the gesture from the module. May lead to removal of some primitives (if the primitive is not used by any gestures anymore)

Parameters

gesture_name (str) – name of the gesture to remove

get_currently_used_primitives() Set[str]

Get names of all primitives that are used by any of the available gestures.

Returns

names of currently used primitives

Return type

Set[str]

reset() None

Resets all position trackers

classmethod calibrate() None
classmethod pre_initialize() None

Can be called to perform all of the time consuming setup of the Module before initialization. If is not called then all the setup is done when the Module is first initialized.

Author: Carmen Meinson

class RawData

Bases: object

add_landmark(bodypart_name: str, landmark_name: str, coordinates: numpy.ndarray) None
combine(raw_data: scripts.core.raw_data.RawData) None

Combine the data from 2 RawData instances. Note the instances MUST contain different body parts.

Parameters

raw_data (RawData) – RawData instance the data of which should be copied over to this one

Raises

RuntimeError – raised if the instances do contain the same body parts

get_bodyparts() Set[str]
Returns

names of all the body parts that have been added to the RawData instance

Return type

Set[str]

get_data(bodypart_name: str) Optional[Dict[str, numpy.ndarray]]

Returns the data of the body part, if it has been added (aka if it has been detected in the frame)

Parameters

bodypart_name (str) – bodypart, the data of which should be returned. e.g. “body” or “Left”

Returns

Dictionary of all the landmarks added to the specific body part with the name of the landmark as key and the coordinates [x,y(,z)] as value

Return type

Optional[Dict[str, np.ndarray]]

get_landmark(bodypart_name: str, landmark_name: str) Optional[numpy.ndarray]

Returns the coordinates of the landmark, if it has been added to the specified body part (aka if it has been detected in the frame)

Parameters
  • bodypart_name (str) – body part that the specified landmark should belong to e.g. “body” or “Left”

  • landmark_name (str) – landmark, the coordinate of which should be returned. e.g. “index_tip” or “wrist”

Returns

coordinate of the landmark [x,y(,z)]

Return type

Optional[np.ndarray]

Author: Carmen Meinson

class Position(raw_data: Dict[str, numpy.ndarray], used_primitives: Optional[Set[str]] = None)

Bases: object

get_primitive(name: str) Optional[bool]

Returns the state of the given primitive in the Position if it has been calcualated. Returns None If the primitive has not been calculated, hence if it either is not defined for current module or the land marks needed to calculate it were not provided

Parameters

name (str) – name of the primitive (e.g. “palm_facing_camera”, “index_pinched” or “punchleft”)

Returns

state of the primitive

Return type

Optional[bool]

get_landmark(name: str) Optional[numpy.ndarray]

Returns the coordinates of the landmark. Returns None if the landmarks was not provided to the Position on initialization

Parameters

name (str) – name of the landmark

Returns

coordinates of the landmark [x,y(,z)]

Return type

Optional[np.ndarray]

get_primitives_names() Set[str]
Returns

Names of all primitives that are calculated by the specific modules Position class on initialization

Return type

List[str]

Author: Carmen Meinson

class PositionTracker(name: str, position_class: Type[scripts.core.position.Position])

Bases: object

update(raw_data: scripts.core.raw_data.RawData, used_primitives: Optional[Set[str]] = None) Set[str]

Creates a Position instance out of the provided RawData and compares the state of all the primitives to the previous Position. Returns the list of all the changed primitives. If there was no previous position (so if it is the first update() call or if the tracker has been reset) all primitives are considered to have changed.

If used_primitives is not None the Position class may only calculate the primitives given. If it is None all primitives are calculated

Parameters
  • raw_data (RawData) – RawData instance containing all the landmarks coordinates detected in the frame

  • used_primitives (Set[str]) – names of all the primitives that are currently used by the module

Returns

set of changed primitives since the last update() call

Return type

Set[str]

get_name() str

returns the name of the tracker thus also reflecting the name of the body part

get_current_position() Optional[scripts.core.position.Position]

Returns the Position instance generated by the last update() call. If there has been no such call or if the tracker has been reset returns None

Returns

last Position instance

Return type

Optional[Position]

reset() Set[str]

Resets the Tracker instance to the initial state and returns all the changed primitives in doing so. Thus the current position is again None and by default all primitives are considered to have changed (unless the tracker was already in the reset state, in which case none of the primitives are considered to have changed)

Returns

set of changed primitives

Return type

Set[str]

Author: Carmen Meinson

class Primitive(name, state)

Bases: tuple

name

Alias for field number 0

state

Alias for field number 1

class GestureFactory(name: str, gesture_class: Type[scripts.core.gesture.Gesture], primitives: Set[scripts.core.gesture_factory.Primitive])

Bases: object

update(tracker: scripts.core.position_tracker.PositionTracker) Optional[scripts.core.gesture.Gesture]

Creates an instance of the gesture it represents if the current position in the provided tracker matches the gesture criteria (if all the primitives of interest are in needed states)

Parameters

tracker (PositionTracker) – tracker containing the current position to be checked

Returns

a new gesture instance if such is created

Return type

Optional[Gesture]

get_name() str

Author: Carmen Meinson

class Gesture(name: str, position_checker: Callable[[scripts.core.position.Position], bool], tracker: scripts.core.position_tracker.PositionTracker, max_pos_q_size: Optional[int] = None)

Bases: object

update() bool

Records the current Position of the body part this Gesture was created from. With that Position checks if the Gesture still remains active (if all the states of the primitives of interest are in the required states) (This function is intended to be called each frame until the Gesture deactivates)

Returns

The current state of the Gesture

Return type

bool

get_name() str

Returns name of represented gesture.

Returns

name of represented gesture

Return type

str

get_bodypart_name() str

Returns name of the body part of the represented gesture.

Returns

name of body part

Return type

str

get_frames_held() int

Note the number of positions stored and retrievable from the gesture instance may be less then the number of frames held in case the max_pos_q_size has been reached.

Returns

number of frames the gesture as been held for

Return type

int

get_frames_stored() int

Note the number of positions stored and retrievable from the gesture instance may be less then the number of frames held in case the max_pos_q_size has been reached.

Returns

number of positions stored in the gesture instance

Return type

int

get_last_position() scripts.core.position.Position
get_last_n_positions(n: int) List[scripts.core.position.Position]
get_time_held() float

Author: Carmen Meinson

class GestureEvent(gesture_types: Set[str], event_trigger_types: Set[str], bodypart_types: Set[str])

Bases: object

set_trigger_functions(funcs: Dict[str, Callable]) None
set_bodypart_names(bodypart_names) None
set_up() None

Called right after the event is added to the model - so after the trigger functions and bodypart names have been set. The method could be used to for example set up the display elements.

update() None

Checks the event firing conditions and then may call according trigger functions. (the function is expected to be called each frame until the state of the GestureEvent becomes False)

get_state() bool
force_deactivate() None

As some events need to finish some processing before being deactivated, they can implement this method. It is called when the events are removed from the model. (e.g. for example if an event has called the mouse press function it needs to release the mouse before deactivation)

notify_gesture_activated(gesture: scripts.core.gesture.Gesture) None

Notifies that a new gesture of the same type that is used by the GestureEvent has activated. If the activated gesture has the correct body part name and gesture type combination, it is stored in the GestureEvent instance. The state of the GestureEvent is then checked in case the activation of the gesture changed it.

Parameters

gesture (Gesture) – the gesture that activated

notify_gesture_deactivated(gesture: scripts.core.gesture.Gesture) None

Notifies that a new gesture of the same type that is used by the GestureEvent has deactivated. If the deactivated gesture was previously stored in the GestureEvent instance it is now removed (although there may be cases where it is kept to perform calculations later). The state of the GestureEvent is then checked in case the deactivation of the gesture changed it.

Parameters

gesture (Gesture) – the gesture that deactivated

get_all_used_gestures() Set[str]
get_event_trigger_types() Set[str]
get_bodypart_types() Set[str]

Module contents

stuff text