Category: Blog

  • require-label-prefix

    GitHub Action – Require Label Prefix

    CI

    Use this GitHub action to either warn or add a default label based on prefixes.

    Usage

    steps:
      - uses: trstringer/require-label-prefix@v1
        with:
          secret: ${{ github.TOKEN }}
    
          # prefix is set to whatever prefix you are trying to enforce. For
          # instance, if you want to make sure size labels (e.g. "size/S", "size/L")
          # are enforced, the prefix would be "size".
          prefix: size
    
          # The prefix is divided by the suffix by some separator. This defaults
          # to "https://github.com/" and is typically this, but it could be anything (e.g. ":").
          # labelSeparator: "https://github.com/"
    
          # addLabel, when set to "true", will label the issue with defaultLabel if
          # the issue doesn't have a label with the prefix. If this is set to "false"
          # then a label won't be added, there will just be a comment requesting that
          # somebody adds a label with the labelPrefix.
          # Options: "true", "false" (default).
          # addLabel: false
    
          # If addLabel is set, defaultLabel is the label that will be added if there
          # is no label with this prefix already on the issue. E.g. "size/needed".
          # defaultLabel: "size/needed"
    
          # If you want to only comment on or label issues that are part of a milestone
          # then you would set this to "true". Otherwise, all issues are evaluated.
          # Options: "true", "false" (default).
          # onlyMilestone: false

    Visit original content creator repository
    https://github.com/trstringer/require-label-prefix

  • require-label-prefix

    GitHub Action – Require Label Prefix

    CI

    Use this GitHub action to either warn or add a default label based on prefixes.

    Usage

    steps:
      - uses: trstringer/require-label-prefix@v1
        with:
          secret: ${{ github.TOKEN }}
    
          # prefix is set to whatever prefix you are trying to enforce. For
          # instance, if you want to make sure size labels (e.g. "size/S", "size/L")
          # are enforced, the prefix would be "size".
          prefix: size
    
          # The prefix is divided by the suffix by some separator. This defaults
          # to "https://github.com/" and is typically this, but it could be anything (e.g. ":").
          # labelSeparator: "https://github.com/"
    
          # addLabel, when set to "true", will label the issue with defaultLabel if
          # the issue doesn't have a label with the prefix. If this is set to "false"
          # then a label won't be added, there will just be a comment requesting that
          # somebody adds a label with the labelPrefix.
          # Options: "true", "false" (default).
          # addLabel: false
    
          # If addLabel is set, defaultLabel is the label that will be added if there
          # is no label with this prefix already on the issue. E.g. "size/needed".
          # defaultLabel: "size/needed"
    
          # If you want to only comment on or label issues that are part of a milestone
          # then you would set this to "true". Otherwise, all issues are evaluated.
          # Options: "true", "false" (default).
          # onlyMilestone: false

    Visit original content creator repository
    https://github.com/trstringer/require-label-prefix

  • jazz

    Jazz Logo Jazz Serverless Development Platform

    Build Status License GitHub release Gitter Slack Chat

    Seamlessly build, deploy & manage cloud-native serverless applications!

    Introduction

    Jazz, a serverless development platform, accelerates adoption of serverless technology within your enterprise. Jazz comes with a beautiful UI that lets developers quickly create serverless applications with a click of a button. Its modular design makes it easy to add new integrations that your enterprise needs.

    Overview

    • Services – As of today, Jazz can help build functions, APIs and static websites. Fully customizable template-based design makes it easy for admins to define new ones and expose them as services to the developers. Jazz also supports Serverless Framework and developers can leverage its rich ecosystem to build serverless apps. Support for AWS SAM will be coming soon!
    • Deployment Targets – Currently, Jazz can deploy to AWS & Azure. Developers can select their target cloud provider during service creation and Jazz will orchestrate the provisioning in the respective public cloud. GCP support will be available in the near future.
    • CI/CD – Jazz comes with CI/CD by default. It creates a code repository for each service, assigns appropriate permissions and adds a web hook to trigger build/deployment workflows whenever it sees a commit. We leverage Jenkins open source for the build process and Serverless Framework for deploying these services.
    • One-Click Development Environments – Jazz automatically creates a new dedicated development environment for each git branch in the service repository. This allows parallel development in a multi-developer team without a hitch! Jazz takes care of deleting these environments once the branch gets deleted.
    • Security Controls – Centralized platform allows administrators to define & apply security controls from a single place through Jazz. Admins can choose to enforce the controls on every service that gets created using Jazz allowing them to make them secure by default. Many best practices like applying the principle of least privilege, code (& dependency) scans during CI/CD, preventing default public access are available by default in Jazz.
    • Secret Management through T-Vault – Jazz comes with T-Vault integration by default encouraging developers to leverage secrets management service for storing their secrets. Each service gets its own set of safes (one per environment) and functions (AWS Lambda only) that are part of the service get automatically configured with required permissions to retrieve the secrets. Developers can simply login into T-Vault interface and store their secrets in the allocated safes. No more hard-coded secrets in the code!
    • Extensions – Jazz is designed to integrate and work well with other systems that your enterprise needs. You can check out our optional extensions like Azure, Slack, Splunk, Apigee etc. Centralized configuration helps Jazz admins to easily enable/disable these features as per their needs.
    • Multiple Accounts & Regions – Jazz’s multi account/region extension allows you to add support for multiple accounts & regions to an existing Jazz stack. Developers can simply select the target account & region from the UI to deploy their services. Admins can add/remove new accounts/regions to the stack with ease.
    • Other Features – Jazz comes with many useful integrations like SCM (Gitlab/Bitbucket), monitoring (CloudWatch), logging (ElasticSearch), authentication (Cognito), access controls, code quality metrics (SonarQube) by default.
    • Utilities – Jazz comes with many utilties that allow admins to monitor, govern and report the usage of serverless services through Jazz environment.

    Jazz is open-sourced and under active development by T-Mobile’s Cloud Center of Excellence.

    Watch the video preview here.

    User Guide

    For complete user guide, see our wiki.

    Architecture

    Following is the high level logical architecture of Jazz.

    Jazz Architecture

    Installation

    You can install Jazz in your AWS account using the automated installer.

    Try Jazz!

    You can try out public preview version of Jazz by registering with your email address here. You will need a registration code which can be requested by joining slack.

    Platform Development

    Branching/Release Strategy

    1. Breaking/nontrivial features first go into named feature branches cut from develop
    2. When/if a feature branch is chosen to be included in the next release, it is merged into develop
    3. Release testing happens in develop
    4. When confirmed/vetted, develop is merged into master, and master becomes the current release.
    5. Small fixes explicitly intended for the next release can be PRed directly into develop without first needing a feature branch.

    tl;dr master is always the current release, develop is always the current state of the next release. If you want to contribute a PR, we recommend you fork and work in a branch off of develop, then PR against develop. Project owners will move you into a feature branch if they deem it necessary.

    License

    Jazz is released under the Apache 2.0 License

    Visit original content creator repository https://github.com/tmobile/jazz
  • coverage-helper

    coverage-helper

    crates.io docs.rs license msrv github actions maintenance-status

    Note: This crate is now deprecated in favor of the pattern that is recommended in the cargo-llvm-cov documentation.

    If you want to ignore all #[test]-related code, you can use module-level #[coverage(off)] attribute:

    #![cfg_attr(coverage_nightly, feature(coverage_attribute))]
    
    #[cfg(test)]
    #[cfg_attr(coverage_nightly, coverage(off))]
    mod tests {
        // ...
    }

    cargo-llvm-cov excludes code contained in the directory named tests from the report by default, so you can also use it instead of #[coverage(off)] attribute.


    Helper for taiki-e/cargo-llvm-cov#123.

    Note: coverage-helper 0.2 supports #[coverage(off)]. See coverage-helper 0.1 for versions that support #[no_coverage].

    Usage

    Add this to your Cargo.toml:

    [dev-dependencies]
    coverage-helper = "0.2"

    Examples

    use coverage_helper::test;
    
    #[test]
    fn my_test() {
        // ...
    }

    Expanded to:

    #[cfg_attr(all(coverage_nightly, test), coverage(off))]
    #[::core::prelude::v1::test]
    fn my_test() {
        // ...
    }

    License

    Licensed under either of Apache License, Version 2.0 or MIT license at your option.

    Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

    Visit original content creator repository https://github.com/taiki-e/coverage-helper
  • openhistoricalmap-embed

    Embeddable OpenHistoricalMap

    This is a simple world historical map based on OpenHistoricalMap data that is suitable for embedding inside a third-party webpage. Using URL parameters, you can choose a location, time period, and cartographic style. To show a place’s evolution, you can also animate smoothly between two arbitrary dates. The map responds to gestures interactively, but the dates can only be adjusted via the URL.

    Visit the map directly at embed.openhistoricalmap.org. Or export a code snippet for your webpage by opening the Share panel on the right side of OHM’s homepage and changing the format to “HTML”.

    URL parameters

    Viewport

    The URL has typical z/x/y parameters for map zoom and center.

    So a parameter like #map=10/43.9367/12.5528 is zoom 10 showing San Marino in Italy, which is at 43.9367/12.5528 in lon,lat format of decimal degrees. See the map.

    An embedded map is typically of a different size and aspect ratio from the original and thus must be scaled in order to cover a comparable area. This is accomplished by passing the original map’s bounding box in the hash as &bbox=minlon,minlat,maxlon,maxlat. Once the embedded map gets its initial framing from the bbox the normal hash mechanism takes over. The San Marino example could be bounded by appending &bbox=12.321338653564453,43.86782687726672,12.58037567138672,44.008373185063874 to the URL. See this map.

    Projection

    The projection parameter accepts the following values:

    • mercator, Web Mercator projection, represents the world as a square. It allows you to view both hemispheres at the same time but with extreme area distortion at the upper latitudes. This is the default projection.
    • vertical-perspective, Vertical Perspective projection, resembles a globe. It avoids area distortion but only shows one hemisphere at a time.
    • globe is a hybrid of vertical-perspective at low zoom levels and mercator at high zoom levels where the Web Mercator projection’s distortions matter much less.

    The 🌐 button at the upper-left corner toggles between mercator and globe projection.

    Dates

    Without a date parameter, the map shows everything in the OHM tiles for which there is a style specified.

    OHM-specific parameters:

    • date is a valid YYYY, YYYY-MM, or YYYY-MM-DD date, but we allow the year to be a variable number of digits or negative, unlike ISO 8601-1. So here is San Marino in the year 1500 #map=10/43.9367/12.5528&date=1500. See this map.
    • layer selects one of the OHM-compatible styles currently offered on openhistoricalmap.org:

    Map layer layer Example
    Historical O New York in 1700
    Railway R Sydney in 1924
    Woodblock W San Marino in 1500
    Japanese Scroll J Osaka in 1970

    The map can optionally animate if you specify the following parameters:

    • start_date is the initial value of date at the beginning of the animation. If you also specify date, the start_date is ignored in favor of date.
    • end_date is the final value of date at the end of the animation.
    • interval is the difference in the dates depicted by any two consecutive frames of the animation, expressed as an ISO 8601-1 duration. For example, P10Y6M1D advances each frame by 10 years, 6 months, and 1 day, while -P10Y6M1D turns back the clock by 10 years, 6 months, and 1 day on each frame. This parameter only supports years, months, and/or days. By default, the animation advances by one year at a time.
    • framerate is the frequency of the animation measured in hertz, defaulting to 1 (1 hertz, or 1 frame per second).

    Language

    By default, map labels appear in your preferred language according to your browser preferences. You can also override this preference by setting the language parameter to an ISO 639 language code. For example, add &language=grc&date=-0999 to see Middle Babylon labeled in Ancient Greek or &language=la&date=-0999 to see it in Latin instead of the contemporary cuneiform. If OHM doesn’t have the name of a place in this preferred language, the label appears in the contemporary local language as a last resort. To force the display of names in contemporary local languages, set the language parameter to mul (the ISO 639 code for multilingual content).

    Embedding

    Simply use code like this to embed:

        <iframe src="https://embed.openhistoricalmap.org/#map=10/43.9367/12.5528&date=1500&layer=O" height="500" width="100%" title="OpenHistoricalMap: San Marino in 1500"></iframe> 

    Here’s an example iframe.

    Feedback

    Please submit bug reports and feature requests to OpenHistoricalMap’s central issue tracker, noting “OpenHistoricalMap embed” somewhere in the title or description.

    Visit original content creator repository
    https://github.com/OpenHistoricalMap/openhistoricalmap-embed

  • CRNN_Tensorflow

    CRNN_Tensorflow

    This is a TensorFlow implementation of a Deep Neural Network for scene text recognition. It is mainly based on the paper “An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition”. You can refer to the paper for architecture details. Thanks to the author Baoguang Shi.

    The model consists of a CNN stage extracting features which are fed to an RNN stage (Bi-LSTM) and a CTC loss.

    Installation

    This software has been developed on Ubuntu 16.04(x64) using python 3.5 and TensorFlow 1.15. Since it uses some recent features of TensorFlow which is incompatible with older versions.

    The following methods are provided to install dependencies:

    Conda

    You can create a conda environment with the required dependencies using:

    conda env create -f crnntf-env.yml
    

    Pip

    Required packages may be installed with

    pip3 install -r requirements.txt
    

    Testing the pre-trained model

    Evaluate the model on the synth90k dataset

    In this repo you will find a model trained on the Synth 90k dataset. When the tfrecords file of synth90k dataset has been successfully generated you may evaluated the model by the following script

    The pretrained crnn model weights on Synth90k dataset can be found here

    python tools/evaluate_shadownet.py --dataset_dir PATH/TO/YOUR/DATASET_DIR 
    --weights_path PATH/TO/YOUR/MODEL_WEIGHTS_PATH
    --char_dict_path PATH/TO/CHAR_DICT_PATH 
    --ord_map_dict_path PATH/TO/ORD_MAP_PATH
    --process_all 1 --visualize 1
    

    If you set visualize true the expected output during evaluation process is

    evaluate output

    After all the evaluation process is done you should see some thing like this:

    evaluation_result

    The model’s main evaluation index are as follows:

    Test Dataset Size: 891927 synth90k test images

    Per char Precision: 0.974325 without average weighted on each class

    Full sequence Precision: 0.932981 without average weighted on each class

    For Per char Precision:

    single_label_accuracy = correct_predicted_char_nums_of_single_sample / single_label_char_nums

    avg_label_accuracy = sum(single_label_accuracy) / label_nums

    For Full sequence Precision:

    single_label_accuracy = 1 if the prediction result is exactly the same as label else 0

    avg_label_accuracy = sum(single_label_accuracy) / label_nums

    Part of the confusion matrix of every single char looks like this:

    evaluation_confusion_matrix

    Test the model on the single image

    If you want to test a single image you can do it with

    python tools/test_shadownet.py --image_path PATH/TO/IMAGE 
    --weights_path PATH/TO/MODEL_WEIGHTS
    --char_dict_path PATH/TO/CHAR_DICT_PATH 
    --ord_map_dict_path PATH/TO/ORD_MAP_PATH
    

    Test example images

    Example test_01.jpg

    Example image1

    Example test_02.jpg

    Example image2

    Example test_03.jpg

    Example image3

    Training your own model

    Data preparation

    Download the whole synth90k dataset here And extract all th files into a root dir which should contain several txt file and several folders filled up with pictures. Then you need to convert the whole dataset into tensorflow records as follows

    python tools/write_tfrecords 
    --dataset_dir PATH/TO/SYNTH90K_DATASET_ROOT_DIR
    --save_dir PATH/TO/TFRECORDS_DIR
    

    During converting all the source image will be scaled into (32, 100)

    Training

    For all the available training parameters, check global_configuration/config.py, then train your model with

    python tools/train_shadownet.py --dataset_dir PATH/TO/YOUR/TFRECORDS
    --char_dict_path PATH/TO/CHAR_DICT_PATH 
    --ord_map_dict_path PATH/TO/ORD_MAP_PATH
    

    If you wish, you can add more metrics to the training progress messages with --decode_outputs 1, but this will slow training down. You can also continue the training process from a snapshot with

    python tools/train_shadownet.py --dataset_dir PATH/TO/YOUR/TFRECORDS
    --weights_path PATH/TO/YOUR/PRETRAINED_MODEL_WEIGHTS
    --char_dict_path PATH/TO/CHAR_DICT_PATH --ord_map_dict_path PATH/TO/ORD_MAP_PATH
    

    If you has multiple gpus in your local machine you may use multiple gpu training to access a larger batch size input data. This will be supported as follows

    python tools/train_shadownet.py --dataset_dir PATH/TO/YOUR/TFRECORDS
    --char_dict_path PATH/TO/CHAR_DICT_PATH --ord_map_dict_path PATH/TO/ORD_MAP_PATH
    --multi_gpus 1
    
    

    The sequence distance is computed by calculating the distance between two sparse tensors so the lower the accuracy value is the better the model performs. The training accuracy is computed by calculating the character-wise precision between the prediction and the ground truth so the higher the better the model performs.

    Tensorflow Serving

    Thanks for Eldon’s contribution of tensorflow service function:)

    Since tensorflow model server is a very powerful tools to serve the DL model in industry environment. Here’s a script for you to convert the checkpoints model file into tensorflow saved model which can be used with tensorflow model server to serve the CRNN model. If you can not run the script normally you may need to check if the checkpoint file path is correct in the bash script.

    bash tfserve/export_crnn_saved_model.sh
    

    To start the tensorflow model server you may check following script

    bash tfserve/run_tfserve_crnn_gpu.sh
    

    There are two different ways to test the python client of crnn model. First you may test the server via http/rest request by running

    python tfserve/crnn_python_client_via_request.py ./data/test_images/test_01.jpg
    

    Second you may test the server via grpc by running

    python tfserve/crnn_python_client_via_grpc.py
    

    Experiment

    The original experiment run for 2000000 epochs, with a batch size of 32, an initial learning rate of 0.01 and exponential decay of 0.1 every 500000 epochs. During training the train loss dropped as follows

    Training loss

    The val loss dropped as follows

    Validation_loss

    2019.3.27 Updates

    I have uploaded a newly trained crnn model on chinese dataset which can be found here. Sorry for not knowing the owner of the dataset. But thanks for his great work. If someone knows it you’re welcome to let me know. The pretrained weights can be found here

    Before start training you may need reorgnize the dataset’s label information according to the synth90k dataset’s format if you want to use the same data feed pip line mentioned above. Now I have reimplemnted a more efficient tfrecords writer which will accelerate the process of generating tfrecords file. You may refer to the code for details. Some information about training is listed bellow:

    image size: (280, 32)

    classes nums: 5824 without blank

    sequence length: 70

    training sample counts: 2733004

    validation sample counts: 364401

    testing sample counts: 546601

    batch size: 32

    training iter nums: 200000

    init lr: 0.01

    Test example images

    Example test_01.jpg

    Example image1

    Example test_02.jpg

    Example image2

    Example test_03.jpg

    Example image3

    training tboard file

    Training loss

    The val loss dropped as follows

    Validation_loss

    2019.4.10 Updates

    Add a small demo to recognize chinese pdf using the chinese crnn model weights. If you want to have a try you may follow the command:

    cd CRNN_ROOT_REPO
    python tools/recongnize_chinese_pdf.py -c ./data/char_dict/char_dict_cn.json 
    -o ./data/char_dict/ord_map_cn.json --weights_path model/crnn_chinese/shadownet.ckpt 
    --image_path data/test_images/test_pdf.png --save_path pdf_recognize_result.txt
    

    You should see the same result as follows:

    The left image is the recognize result displayed on console and the right image is the origin pdf image.

    recognize_result_console

    The left image is the recognize result written in local file and the right image is the origin pdf image. recognize_result_file

    TODO

    • Add new model weights trained on the whole synth90k dataset
    • Add multiple gpu training scripts
    • Add new pretrained model on chinese dataset
    • Add an online toy demo
    • Add tensorflow service script

    Visitor Count

    Visitor Count

    Acknowledgement

    Please cite my repo CRNN_Tensorflow if you use it.

    Contact

    Scan the following QR to disscuss 🙂 qr

    Visit original content creator repository https://github.com/MaybeShewill-CV/CRNN_Tensorflow
  • ansible-role-moodle

    Ansible role moodle

    Install and configure moodle on your system.

    GitHub Version Issues Pull Requests Downloads
    github Version Issues PullRequests Ansible Role

    This example is taken from molecule/default/converge.yml and is tested on each push, pull request and release.

    ---
    - name: converge
      hosts: all
      become: true
      gather_facts: true
    
      pre_tasks:
        - name: Update apt cache.
          apt: update_cache=yes cache_valid_time=600
          when: ansible_os_family == 'Debian'
          changed_when: false
    
        - name: Check if python3.11 EXTERNALLY-MANAGED file exists
          ansible.builtin.stat:
            path: /usr/lib/python3.11/EXTERNALLY-MANAGED
          register: externally_managed_file_py311
    
        - name: Rename python3.11 EXTERNALLY-MANAGED file if it exists
          ansible.builtin.command:
            cmd: mv /usr/lib/python3.11/EXTERNALLY-MANAGED /usr/lib/python3.11/EXTERNALLY-MANAGED.old
          when: externally_managed_file_py311.stat.exists
          args:
            creates: /usr/lib/python3.11/EXTERNALLY-MANAGED.old
    
        - name: Check if python3.12 EXTERNALLY-MANAGED file exists
          ansible.builtin.stat:
            path: /usr/lib/python3.12/EXTERNALLY-MANAGED
          register: externally_managed_file_py312
    
        - name: Rename python3.12 EXTERNALLY-MANAGED file if it exists
          ansible.builtin.command:
            cmd: mv /usr/lib/python3.12/EXTERNALLY-MANAGED /usr/lib/python3.12/EXTERNALLY-MANAGED.old
          when: externally_managed_file_py312.stat.exists
          args:
            creates: /usr/lib/python3.12/EXTERNALLY-MANAGED.old
    
      roles:
        - role: buluma.moodle

    The machine needs to be prepared. In CI this is done using molecule/default/prepare.yml:

    ---
    - name: prepare
      hosts: all
      become: true
      gather_facts: false
    
      roles:
        - role: buluma.bootstrap
        - role: buluma.buildtools
        - role: buluma.epel
        - role: buluma.mysql
          mysql_databases:
            - name: moodle
              encoding: utf8mb4
              collation: utf8mb4_unicode_ci
          mysql_users:
            - name: moodle
              password: moodle
              priv: "moodle.*:ALL"
        - role: buluma.python_pip
        - role: buluma.openssl
          openssl_items:
            - name: apache-httpd
              common_name: "{{ ansible_fqdn }}"
        - role: buluma.php
        - role: buluma.selinux
        - role: buluma.httpd
          httpd_vhosts:
            - name: moodle
              servername: moodle.example.com
        - role: buluma.cron
        - role: buluma.core_dependencies

    Also see a full explanation and example on how to use these roles.

    The default values for the variables are set in defaults/main.yml:

    ---
    # defaults file for moodle
    
    # The version of moodle to install.
    moodle_version: 401
    
    # A path where to save the data.
    moodle_data_directory: /opt/moodledata
    
    # The permissions of the created directories.
    moodle_directory_mode: "0750"
    
    # Details to connect to the database.
    moodle_database_type: mysqli
    moodle_database_hostname: localhost
    moodle_database_name: moodle
    moodle_database_username: moodle
    moodle_database_password: moodle
    moodle_database_prefix: ""
    
    # The URL where to serve content.
    moodle_wwwroot: "https://{{ ansible_default_ipv4.address }}/moodle"

    The following roles are used to prepare a system. You can prepare your system in another way.

    Requirement GitHub Version
    buluma.bootstrap Ansible Molecule Version
    buluma.buildtools Ansible Molecule Version
    buluma.cron Ansible Molecule Version
    buluma.core_dependencies Ansible Molecule Version
    buluma.epel Ansible Molecule Version
    buluma.httpd Ansible Molecule Version
    buluma.mysql Ansible Molecule Version
    buluma.openssl Ansible Molecule Version
    buluma.php Ansible Molecule Version
    buluma.python_pip Ansible Molecule Version
    buluma.selinux Ansible Molecule Version

    This role is a part of many compatible roles. Have a look at the documentation of these roles for further information.

    Here is an overview of related roles:

    dependencies

    This role has been tested on these container images:

    container tags
    EL 8
    Debian all
    Fedora 38, 39, 40
    opensuse all
    Ubuntu jammy, noble

    The minimum version of Ansible required is 2.12, tests have been done to:

    • The previous version.
    • The current version.
    • The development version.

    If you find issues, please register them in GitHub

    Role History

    Apache-2.0

    Shadow Walker

    Visit original content creator repository https://github.com/buluma/ansible-role-moodle
  • datasets

    Sign Language Datasets

    This repository includes TFDS data loaders for sign language datasets.

    Installation

    From Source

    pip install git+https://github.com/sign-language-processing/datasets.git

    PyPi

    pip install sign-language-datasets

    For apple environments, you may need touse an ARM version of conda:

    # Uninstall Anaconda
    conda activate base
    conda install anaconda-clean
    anaconda-clean --yes
    conda deactivate
    brew uninstall anaconda
    
    # Download and install Miniforge for ARM (Apple Silicon)
    curl -L -O https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
    bash Miniforge3-MacOSX-arm64.sh
    # Follow the instructions
    
    conda create --name=datasets python=3.11
    conda activate datasets
    pip install .
    
    # Test the installation
    python -c "import tensorflow"
    python -c "import sign_language_datasets"

    Usage

    We demonstrate a loading script for every dataset in examples/load.ipynb Open In Colab

    Our config includes the option to choose the resolution and fps, for example:

    import tensorflow_datasets as tfds
    import sign_language_datasets.datasets
    from sign_language_datasets.datasets.config import SignDatasetConfig
    
    # Loading a dataset with default configuration
    aslg_pc12 = tfds.load("aslg_pc12")
    
    # Loading a dataset with custom configuration
    config = SignDatasetConfig(name="videos_and_poses256x256:12",
                               version="3.0.0",  # Specific version
                               include_video=True,  # Download and load dataset videos
                               process_video=True,  # Process videos to tensors, or only save path to video
                               fps=12,  # Load videos at constant, 12 fps
                               resolution=(256, 256),  # Convert videos to a constant resolution, 256x256
                               include_pose="holistic")  # Download and load Holistic pose estimation
    rwth_phoenix2014_t = tfds.load(name='rwth_phoenix2014_t', builder_kwargs=dict(config=config))

    Datasets

    Dataset Videos Poses Versions
    aslg_pc12 N/A N/A 0.0.1
    asl-lex No 2.0.0
    rwth_phoenix2014_t Yes Holistic 3.0.0
    autsl Yes OpenPose, Holistic 1.0.0
    dgs_corpus Yes OpenPose, Holistic 3.0.0
    dgs_types Yes 3.0.0
    how2sign Yes OpenPose 1.0.0
    sign2mint Yes 1.0.0
    signtyp Links 1.0.0
    swojs_glossario Yes 1.0.0
    SignBank N/A 1.0.0
    wlasl Failed OpenPose None
    wmtslt Yes OpenPose, Holistic 1.2.0
    signsuisse Yes 1.0.0
    msasl None
    Video-Based CSL None
    RVL-SLLL ASL None
    ngt_corpus Yes 3.0.0
    bsl_corpus No No 3.0.0

    Data Interface

    We follow the following interface wherever possible to make it easy to swap datasets.

    {
        "id": tfds.features.Text(),
        "signer": tfds.features.Text() | tf.int32,
        "video": tfds.features.Video(shape=(None, HEIGHT, WIDTH, 3)),
        "depth_video": tfds.features.Video(shape=(None, HEIGHT, WIDTH, 1)),
        "fps": tf.int32,
        "pose": {
            "data": tfds.features.Tensor(shape=(None, 1, POINTS, CHANNELS), dtype=tf.float32),
            "conf": tfds.features.Tensor(shape=(None, 1, POINTS), dtype=tf.float32)
        },
        "gloss": tfds.features.Text(),
        "text": tfds.features.Text()
    }

    Adding a new dataset

    For general instructions, see the TFDS guide to writing custom datasets. Instructions below are specific to this repository.

    Make a new folder inside sign_language_datasets/datasets with the same name as the dataset. As a convention, the name of the dataset should be lowercase and words should be separated by an underscore. Example:

    cd sign_language_datasets/datasets
    tfds new new_dataset

    For our purposes, creating a custom TFDS dataset means writing a new class which inherits from tfds.core.GeneratorBasedBuilder. If you use tfds new to create a new dataset then the dataset class is stored in a file with the exact same name as the dataset, i.e. new_dataset.py. new_dataset.py must contain a line similar to:

    class NewDataset(tfds.core.GeneratorBasedBuilder):

    Registering a new dataset

    The mechanism to add a custom dataset to TFDS’ dataset registry is to import the class NewDataset. For this reason the folder sign_language_datasets/datasets/new_dataset must have an __init__.py file that imports the class NewDataset:

    from .new_dataset import NewDataset

    Even though the name of the class is NewDataset, it will be available for loading in lowercase and uppercase characters are interpreted as the start of a new word that should be separated with an underscore. This means that the class can be loaded as follows:

    ds = tfds.load('new_dataset')

    Generating checksums

    The folder for the new dataset should contain a file checksums.tsv with checksums for every file in the dataset. This allows the TFDS download manager to check the integrity of the data it downloads. Use the tfds build tool to generate the checksum file:

    tfds build --register_checksums new_dataset.py

    Use a dataset configuration which includes all files (e.g. does include the video files if any) using the --config argument. The default behaviour is to build all configurations which might be redundant.

    Why not Huggingface Datasets?

    Huggingface datasets do not work well with videos. From the lack of native support of the video type, to lack of support of arbitrary tensors. Furthermore, they currently have memory leaks that prevent from saving even the smallest of video datasets.

    Cite

    @misc{moryossef2021datasets, 
        title={Sign Language Datasets},
        author={Moryossef, Amit and M\"{u}ller, Mathias},
        howpublished={\url{https://github.com/sign-language-processing/datasets}},
        year={2021}
    }
    Visit original content creator repository https://github.com/sign-language-processing/datasets
  • Paper-Geometric-Mechanics-of-Contact-Switching-Systems

    Paper-Geometric-Mechanics-of-Contact-Switching-Systems

    This is the code repository for the paper “Geometric Mechanics of Contact-Switching Systems”: <10.1109/LRA.2023.3327930>. This repository can generate the plots provided in the paper.

    Information on using this repository

    Each file within the “Code” folder is intended to be run sectionwise. For more details on the terminology and techniques, please refer to the paper linked above.

    1. The stratified panels shown in the paper (in Fig.1(c), Fig.3(d), Fig.3(e), Fig.4(b), and Fig.6) are generated in the “EfficiencyAndPanels.m” code.

    2. The contact interpolation Fig.3(a) is generated in the “ContactInterpolationFunction.m” code.

    3. The local connection vector fields and the constraint curvature functions shown in figures 3(b) and 3(c) are generated using the GeometricSystemPlotter repository developed by collaborators at OSU. A link to this repository can be found here: https://github.com/OSU-LRAM/GeometricSystemPlotter. The code for the class of systems explored in this paper is provided in the “GeometricSystemPlotter” folder in the exact same structure of the GeometricSystemPlotter repository. So, if you’re interested in testing things out on the systems used in this paper, please clone the repository from that link and then copy the contents of the “GeometricSystemPlotter” folder into their corresponding location.

    4. The shape change or gait trajectory in Fig.4(a) is generated in the “ContactInterpolationFunction.m” code.

    5. All animations provided with this paper can be recreated using the “Animations_HybridContact.m” code.

    6. The main video for this paper can be found in the “Video” folder. This video is intended to be a condensed illustrative aid to the paper (with less jargon).

    Visit original content creator repository
    https://github.com/Animal-Inspired-Motion-And-Robotics-Lab/Paper-Geometric-Mechanics-of-Contact-Switching-Systems

  • 2FA-Auth

    2FA-Auth


    2FA


    2FA-Auth is a BASH script that provides a user-friendly way to generate “Two-Factor Authentication (2FA)” code. It works like Google Authenticator® and similar programs, but you can use your GNU/Linux terminal, instead of your cellphone.

    For more information, look for “Two-Factor Authentication” in sites or forums.


    System Requirement

    • GNU/Linux distribution with BASH interpreter

    • GIT — used to clone 2FA-Auth into your computer

    • GnuPG — used to keep your site/service token encrypted

    • OATH Toolkit a.k.a. OATHTOOL — used to generate 2FA codes

    2FA-Auth can automatically install GnuPG and OAth Toolkit, but if it fails, please check how to install the programs above, according to your distribution.

    This automatic method of package installation involves these package managers:

    • APT and APT-GET for Debian-based systems
    • DNF, URPMI and YUM for RedHat-based systems
    • EMERGE and EQUO for Gentoo-based systems
    • PACMAN for Arch-based systems
    • ZYPPER for SUSE/openSUSE-based systems

    The next step is that you MUST create or import GPG Keys in your profile. They are used by GnuPG to encrypt and decrypt your tokens.


    PDF FILE: “2FA-Auth/doc/How_to_use_2FA-Auth.pdf”

    This file explain how to use 2FA-Auth.

    It’s a simple “how to use” manual, but it helps you A LOT!


    Where does 2FA-Auth save my tokens and GnuPG ID?

    GPG key is created (or imported) into $HOME/.gnupg/ while GPG ID is part of your key (UserID) and 2FA-Auth asks for it. Your UserID is the e-mail used to create your key.

    2FA-Auth saves your 2FA token(s) in this directory: $HOME/.config/2fa-auth/ and your GnuPG UserID is saved in $HOME/.config/2fa-auth/2fa-auth.info


    Vinicius de Alencar (alencc1986) – 2021
    Visit original content creator repository https://github.com/alencc1986-zz/2FA-Auth