Category: Blog

  • LABORATORY-PASSPORT

    LABORATORY-PASSPORT

    I discover few weeks ago the library Passport.js that you can find at this URL : http://www.passportjs.org/

    It makes multiple Authentication through google, facebook, twitter and so on easy using just the ClientId and ClientSecret of the different platform.

    It becomes a must have in my toolbox for managing this kind of challenge.

    Plan

    1. How to use Passport.js
    2. How to create clientID and clientSecret for facebook
    3. How to create clientID and clientSecret for google

    How to use Passport.js

    1. Install Passport.js
    $ npm install Passport.js
    

    In the Express server, use :

    const passport = require('passport');
    
    app.use(passport.initialize());
    app.use(passport.session());
    
    passport.serializeUser(function (user, cb) {
      cb(null, user);
    });
    
    passport.deserializeUser(function (obj, cb) {
      cb(null, obj);
    });
    
    1. Install the dependencies depending of the passport we need

    facebook

    $ npm install passport-facebook
    

    google

    $ npm install passport-google-oauth
    
    1. Enable the Passport depending of the passport

    facebook

    const FacebookStrategy = require('passport-facebook').Strategy;
    
    passport.use(new FacebookStrategy({
        clientID: config.facebookAuth.clientID,
        clientSecret: config.facebookAuth.clientSecret,
        callbackURL: config.facebookAuth.callbackURL
      }, function (accessToken, refreshToken, profile, done) {
        return done(null, profile);
      }
    ));
    

    google

    const GoogleStrategy = require('passport-google-oauth').OAuth2Strategy;
    
    passport.use(new GoogleStrategy({
        clientID: config.googleAuth.clientID,
        clientSecret: config.googleAuth.clientSecret,
        callbackURL: config.googleAuth.callbackURL
      }, function (accessToken, refreshToken, profile, done) {
        return done(null, profile);
      }
    ));
    
    1. Add the ClientID and ClientSecret inside the config.js (see below how to get them)

    2. Create the route for getting the information out of the Authentication

    The scope depend of the Strategy (facebook or google…) and can be find on the documentation of the strategy (google documentation or facebook documentation…)

    facebook

    router.get('/profile', isLoggedIn, function (req, res) {
      console.log(req.user)
    });
    
    router.get('/auth/facebook', passport.authenticate('facebook', {
      scope: ['public_profile', 'email']
    }));
    
    router.get('/auth/facebook/callback',
      passport.authenticate('facebook', {
        successRedirect: '/profile',
        failureRedirect: '/error'
      })
    );
    

    google

    router.get('/profile_google', isLoggedIn, function (req, res) {
      console.log(req.user)
    });
    
    router.get('/auth/google', passport.authenticate('google', {
      scope: ['profile', 'email']
    }));
    
    router.get('/auth/google/callback',
      passport.authenticate('google', {
        successRedirect: '/profile_google',
        failureRedirect: '/error'
      })
    );
    

    How to create clientID and clientSecret for facebook

    1. First, connect to the facebook developer console : https://developers.facebook.com/

    Alt text

    1. Click on create a new app and choose the type of app (none in my case)

    Alt text

    1. Add the name to display in the facebook developer interface

    Alt text

    1. Click on facebook login

    Alt text

    1. Click on www since we will be building a website

    Alt text

    1. Since we will be testing in it locally, we will enter the website : http://localhost:3000/

    Alt text

    1. We then arrive on a page where we can find the ClientId (App ID) and the ClientSecret (App Secret) to enter in our config.js file

    Alt text

    How to create clientID and clientSecret for google

    1. First, connect to the google console : https://console.cloud.google.com/

    Alt text

    1. Search in the bar on the top oauth and click on identifiants

    Alt text

    1. Once the page loaded, click on the top create identifiants

    Alt text

    1. In the dropdown, click on ID Client OAuth

    Alt text

    1. Choose the type of application (web application in this case), add a name and dont forget to add the redirection URI at the bottom. Since I am working locally, it will be : http://localhost:3000

    Alt text

    1. You then will get a popup with the ClientID and ClientSecret that you can copy and paste into the config.js file.

    Alt text

    Visit original content creator repository https://github.com/JustalK/LABORATORY-PASSPORT
  • wallpo

    Wallpo

    A powerful image setter for Smart Phone MainScreen and LockScreen for Android

    Setting Wallpaper on Mainscreen

    mainscreen

    Setting Wallpaper on Lockscreen

    lockscrren

    Only three steps to use the wallpo:

    1. Add repository to your build file

         allprojects {
       	repositories {
          	...
      		maven { url 'https://jitpack.io' }
          }
      }
      
    2. Add the dependency

         dependencies {
           implementation 'com.github.sayyedrizwan:wallpo:1.4.0'
       }
      

    OR

    ADD Through MAVEN

    ADD Add repository to your build file

    <repositories>
    	<repository>
    	    <id>jitpack.io</id>
    	    <url>https://jitpack.io</url>
    	</repository>
    </repositories>
    

    Add the maven dependency

    <dependency>
        <groupId>com.github.sayyedrizwan</groupId>
        <artifactId>wallpo</artifactId>
        <version>1.4.0</version>
    </dependency>
    
    1. Add Wallpo Code Where you want to run and set a image as mainscreen wallpaper

       Wallpo.setMainScreenWallpaper(MainActivity.this, imageView, "Wallpaper Set");
       		//imageview is the view where image is loading
      

    OR

    1. Add Wallpo Code Where you want to run and set a image as lockscreen wallpaper

       Wallpo.setLockScreenWallpaper(MainActivity.this, imageView, "LockWallpaper Set"); 
       				//imageview is the view where image is loading
      

    License

    	Copyright 2018-2019 Wallpo Inc.
    
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at
    
    http://www.apache.org/licenses/LICENSE-2.0
    
    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
    

    Founder Rizwan Sayyed

    Visit original content creator repository https://github.com/sayyedrizwan/wallpo
  • moqui-graphql

    Visit original content creator repository
    https://github.com/mkdecisiondev/moqui-graphql

  • DRL_algorithm_library

    Typing SVG

    README

    This is a reinforcement learning algorithm library. The code takes into account both performance and simplicity, with little dependence. The algorithm code comes from spinningup and some researchers engaged in reinforcement learning.

    Algorithms

    The project covers the following algorithms:

    • DQN, Dueling DQN, D3QN
    • DDPG, DDPG-HER, DDPG-PER
    • PPO+GAE, Multi-Processing PPO, Discrete PPO
    • TD3, Multi-Processing TD3
    • SAC
    • MADDPG

    All the algorithms adopt the pytorch framework. All the codes are combined in the easiest way to understand, which is suitable for beginners of reinforcement learning, but the code performance is excellent.

    Reference

    This project also provides the reference of these algorithms:

    • Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
    • CONTINUOUS CONTROL WITH DEEP REINFORCEMENT
    • High-Dimensional Continuous Control Using Generalized Advantage Estimation
    • Proximal Policy Optimization
    • Soft Actor-Critic Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
    • Auto alpha Soft Actor-Critic Algorithms and Applications
    • Addressing Function Approximation Error in Actor-Critic Methods

    Results

    Testing environment: ‘Pendulum-v0’. What you just need to do is running the main.py. Here are the results of several cases:

    spinningup-DDPG reward curve:

    spin_ddpg

    spinningup-TD3 reward curve:

    spin_td3

    spinningup-SAC reward curve:

    spinSAC

    Project tree

    DRL_algorithm_library
    ├─ Arm_env.py
    ├─ DDPG
    │    ├─ DDPG
    │    ├─ DDPG_spinningup
    │    └─ DDPG_spinningup_PER
    ├─ DQN_family
    │    ├─ Agent.py
    │    ├─ __pycache__
    │    ├─ core.py
    │    ├─ main.py
    │    └─ runs
    ├─ MADDPG
    │    ├─ .gitignore
    │    ├─ .idea
    │    ├─ __pycache__
    │    ├─ arguments.py
    │    ├─ enjoy_split.py
    │    ├─ logs
    │    ├─ main_openai.py
    │    ├─ model.py
    │    ├─ models
    │    └─ replay_buffer.py
    ├─ PPO
    │    ├─ .idea
    │    ├─ DiscretePPO
    │    ├─ PPOModel.py
    │    ├─ TrainedModel
    │    ├─ __pycache__
    │    ├─ core.py
    │    ├─ draw.py
    │    ├─ multi_processing_ppo
    │    └─ myPPO.py
    ├─ README.md
    ├─ SAC
    │    ├─ SAC_demo1
    │    └─ SAC_spinningup
    ├─ TD3
    │    ├─ .idea
    │    ├─ Multi-Processing-TD3
    │    ├─ TD3
    │    ├─ TD3_spinningup
    │    └─ TrainedModel
    ├─ __pycache__
    │    ├─ Arm_env.cpython-36.pyc
    │    └─ Arm_env.cpython-38.pyc
    ├─ imgs
    │    ├─ spinSAC.png
    │    ├─ spin_ddpg.png
    │    └─ spin_td3.png
    └─ reference
           ├─ 多智能体 MADDPG - Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments - 1706.02275.pdf
           ├─ 强化学习 DDPG - CONTINUOUS CONTROL WITH DEEP REINFORCEMENT 1509.02971.pdf
           ├─ 强化学习 GAE High-Dimensional Continuous Control Using Generalized Advantage Estimation 1506.02438.pdf
           ├─ 强化学习 PPO - Proximal Policy Optimization1707.06347.pdf
           ├─ 强化学习 SAC1 - Soft Actor-Critic Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor - 1801.01290.pdf
           ├─ 强化学习 SAC2 auto alpha  Soft Actor-Critic Algorithms and Applications 1812.05905.pdf
           └─ 强化学习 TD3 - Addressing Function Approximation Error in Actor-Critic Methods 1802.09477.pdf
    

    Requirements

    gym==0.10.5

    matplotlib==3.2.2

    pytorch==1.7.1

    numpy==1.19.2

    Visit original content creator repository https://github.com/ZYunfeii/DRL_algorithm_library
  • Duktape-Windows

    Duktape Windows

    GitHub release (latest by date)

    GitHub issues GitHub stars GitHub All Releases GitHub license

    Method to build Duktape on Windows, and prebuilt binary releases.

    Build Prerequisites

    Install MSYS2

    If build 64-bit Duktape with MINGW64, install x86_64-toolchain

    pacman -S mingw-w64-x86_64-gcc mingw-w64-x86_64-make
    echo "#! /bin/sh" > /mingw64/bin/make
    echo "\"mingw32-make\" \"\$@\"" >> /mingw64/bin/make
    

    If build 32-bit Duktape with MINGW32, install i686-toolchain

    pacman -S mingw-w64-i686-gcc mingw-w64-i686-make
    echo "#! /bin/sh" > /mingw32/bin/make
    echo "\"mingw32-make\" \"\$@\"" >> /mingw32/bin/make
    

    Get Duktape latest release from GitHub release (latest by date)

    Compilation

    tar -xvJf duktape-2.5.0.tar.xz
    cd ./duktape-2.5.0
    make -f Makefile.cmdline DEFINES="-static -s"
    

    Here is enough if you’d like to test only

    Build Windows DLL

    Prerequisites

    If build 64-bit dll with MINGW64, install mingw-w64-x86_64-python2-yaml in MSYS2 without MINGW64 running.

    pacman -S mingw-w64-x86_64-python2-yaml
    

    If build 32-bit dll with MINGW32, install mingw-w64-i686-python2-yaml in MSYS2 without MINGW32 running.

    pacman -S mingw-w64-i686-python2-yaml
    

    Compilation

    python2 tools/configure.py --output-directory ./libduktape --dll
    gcc -shared -o libduktape.dll -static -s -fPIC -Os -pedantic -std=c99 -Wall -Wextra -fstrict-aliasing -fomit-frame-pointer -I./libduktape ./libduktape/duktape.c -lm
    

    Build static library

    gcc -c -fPIC -Os -pedantic -std=c99 -Wall -Wextra -fstrict-aliasing -fomit-frame-pointer ./libduktape/duktape.c
    ar rcs libduktape.a duktape.o
    

    Packaging

    mkdir ./bin
    mv duk.exe ./bin
    mkdir ./lib
    mv libduktape.dll libduktape.a ./lib
    mkdir ./include
    cp -p ./libduktape/duk_config.h ./libduktape/duktape.h ./include
    zip -9 -r $(basename `pwd`)-win$(echo ${MSYSTEM:0-2}).zip ./bin ./include ./lib LICENSE.txt README.rst
    
    Visit original content creator repository https://github.com/mengmo/Duktape-Windows
  • wsl-cloud-init

    wsl-cloud-init

    📝WSL2 上の Ubuntu 24.04 LTS を自動セットアップするスクリプトです。

    このリポジトリには、WSL2 上で Ubuntu 24.04 LTS をセットアップするための PowerShell スクリプト setup-wsl-ubuntu2404-dev.ps1 が含まれています。
    このスクリプトは、cloud-init を利用して開発環境を自動構築します。

    特徴

    • WSL2 上の Ubuntu 24.04 LTS ディストロを自動インストール
    • cloud-init によるプロビジョニングを利用して以下を設定:
      • ユーザーの作成とパスワード設定
      • ロケールとタイムゾーンの設定
      • 必要なパッケージのインストール (例: curl, git, docker.io など)
      • Docker のセットアップ
      • Oh My Posh のインストールと設定
      • WSL の systemd 有効化

    注意事項

    • スクリプト実行前に、同じ名前のディストロが既に存在する場合は自動的に登録解除 (wsl --unregister) が行われます。
    • 登録解除されるディストロのデータは削除されるため、必要に応じて事前にバックアップを取ってください。

    必要な環境

    以下は、WSL のバージョン情報を確認するコマンドの出力例です。

    PS> wsl --version
    WSL バージョン: 2.4.11.0
    カーネル バージョン: 5.15.167.4-1
    WSLg バージョン: 1.0.65
    MSRDC バージョン: 1.2.5716
    Direct3D バージョン: 1.611.1-81528511
    DXCore バージョン: 10.0.26100.1-240331-1435.ge-release
    Windows バージョン: 10.0.26100.3194

    使い方

    1. スクリプトの実行

    .\setup-wsl-ubuntu2404-dev.ps1 -Password "YourPasswordHere"

    2. 初回起動

    スクリプト実行後、自動的に指定したディストロ名(例: ubuntu2404-dev)で WSL の初回起動が行われます。

    3. cloud-init によるプロビジョニングの確認

    セットアップ後、以下のコマンドで cloud-init の状態を確認できます。

    #!/bin/bash
    whoami
    sudo cloud-init status --wait --format json
    sudo cloud-init query userdata
    sudo cat /var/log/cloud-init-output.log

    スクリプトの動作概要

    1. WSL のディストロ名として ubuntu2404-dev を設定。
    2. cloud-init 用の設定ファイルを生成し、必要なパッケージや設定を記述。
    3. WSL に Ubuntu 24.04 をインストール。
      • 既存のディストロ名が存在する場合は登録解除を実行。
    4. インストール完了後に初回起動を実行。
    5. cloud-init によるプロビジョニングを実行。

    参考

    Visit original content creator repository
    https://github.com/MareMare/wsl-cloud-init

  • langgraph-chat-app

    Multi-Agentic Assistant with Advanced RAG Search

    A sophisticated chat application that combines multiple AI agents for flight search, news retrieval, and company financial document analysis. Built with LangGraph, Streamlit, and leveraging advanced RAG (Retrieval Augmented Generation) techniques.

    Demo

    Click for the Demo

    Architecture

    Alt text

    🌟 Key Features

    1. Multi-Agent System

    • Flight Search: Real-time flight information using Amadeus API
    • News Search: Current events and news via Tavily Search API
    • Company Financials: Advanced RAG-based search through company documents

    2. Advanced RAG Implementation

    The system implements a sophisticated multi-stage retrieval pipeline:

    Vector Store Architecture

    • Uses Chroma as the vector database
    • Implements persistent storage for embeddings
    • Automatic PDF processing and chunking
    • Document metadata preservation for source tracking

    Hybrid Search Strategy

    1. Ensemble Retrieval:
      • BM25 (Keyword-based search) – 20%
      • Vector Similarity Search – 40%
      • MMR (Maximal Marginal Relevance) – 40%
    2. Contextual Compression:
      • Cohere re-ranking for result refinement
      • Top-N filtering for most relevant results
      • Citation preservation and source tracking

    🛠 Setup

    Prerequisites

    • Python 3.12+
    • Poetry for dependency management
    • Required API Keys:
      • OpenAI API
      • Tavily Search API
      • Cohere API
      • Amadeus API

    Installation

    1. Clone the repository
    2. Install dependencies:
      poetry install
    3. Copy .env.example to .env and fill in your API keys:

      OPENAI_API_KEY=your_key
      TAVILY_API_KEY=your_key
      CO_API_KEY=your_key
      AMADEUS_CLIENT_ID=your_id
      AMADEUS_CLIENT_SECRET=your_secret
      

    Directory Structure

    ├── company_pdfs/        # Place your company PDFs here
    ├── chroma_db/          # Persistent vector store
    ├── tools/
    │   ├── company_search.py
    │   ├── flight_search.py
    │   └── tavily_search.py
    └── chat.py            # Main application
    

    🚀 Usage

    1. Start the application:

      poetry run streamlit run chat.py
    2. Example queries:

      • Flight Search: “Show me flights from Bangalore to Tokyo”
      • News Search: “What’s happening in AI technology today?”
      • Company Search: “What was Rakuten’s revenue in Q3 2023?”

    🔍 RAG Implementation Details

    Document Processing

    • Recursive character text splitting with 1000-character chunks
    • 200-character overlap between chunks
    • Metadata preservation for source tracking

    Retrieval Pipeline

    1. Initial retrieval using ensemble method:

      • BM25 for keyword matching
      • Dense vector similarity search
      • MMR for diversity in results
    2. Re-ranking:

      • Cohere’s rerank-english-v3.0 model
      • Context-aware compression
      • Top-5 most relevant results

    Citation System

    • Automatic inline citations [1], [2], etc.
    • Source metadata tracking (file name, page number)
    • Expandable source references in UI

    📝 Contributing

    1. Fork the repository
    2. Create your feature branch
    3. Commit your changes
    4. Push to the branch
    5. Create a new Pull Request

    📄 License

    MIT License

    Key LangGraph Concepts

    Nodes

    • Nodes are individual processing units in the application flow, similar to functions in a workflow.
    • Each node (like company_search_node, flight_search_node) handles a specific task and maintains its own state.
    • Example: The company search node processes financial document queries while the flight search node handles travel requests.

    Tools

    • Tools are specialized classes that wrap external APIs or services, making them easily usable within nodes.
    • Each tool (CompanySearchTool, FlightSearchTool, TavilySearchTool) follows a standard interface with invoke() and _run() methods.

    class FlightSearchTool(BaseTool):
        name: str = "flight_search"
        def _run(self, input_str: str) -> str:
            # Process flight search logic
    

    Multiple Nodes and State Management

    • The application uses multiple nodes that work together, each handling different types of queries.
    • State is passed between nodes using a dictionary structure that includes:
      • Messages: Chat history and responses
      • Sources: Retrieved document references
      • Other metadata needed across nodes

    return {
        "messages": past_messages + [new_message],
        "sources": current_sources,
    }
    

    Routing

    • The application uses conditional routing to direct queries to appropriate nodes based on content.
    • Routing decisions are made using:
      • Message content analysis
      • User intent detection
      • Tool availability

    def should_use_company_search(state):
        # Route to company search if query is about financial data
        return "financial" in state.query.lower()
    

    Message Flow

    • User input → Router
    • Router → Appropriate Tool/Node
    • Node processes request
    • Response formatted with citations
    • State updated and returned to user

    This architecture allows for:

    • Modular addition of new capabilities
    • Clear separation of concerns
    • Maintainable state management
    • Flexible routing based on user needs

    Ensemble Retrieval Components:

    BM25 (Okapi BM25)

    • A sophisticated keyword matching algorithm that improves upon traditional word-frequency matching. Think of it as “Ctrl+F” on steroids.
    • It’s smart enough to understand that if a word appears 10 times in a short document, it’s probably more relevant than if it appears 10 times in a very long document. It also prevents common words from dominating the results.

    Vector Similarity Search

    • Converts words and sentences into numerical representations (vectors) where similar meanings are close to each other in mathematical space.
    • For example, “automobile” and “car” would be close together in this space, allowing the system to find relevant content even when exact keywords don’t match.

    Maximal Marginal Relevance (MMR)

    • Ensures search results aren’t repetitive by balancing relevance with diversity.
    • If you have 10 very similar documents that match a query, MMR will pick the best one and then look for other relevant but different perspectives, rather than showing you the same information 10 times.

    Contextual Compression Components:

    Cohere Re-ranking

    • A specialized model that takes the initial search results and re-orders them by actually reading and understanding both the query and the content.
    • Similar to having a human assistant who reads through search results and puts the most relevant ones at the top, but automated.

    Top-N Filtering

    • Takes the re-ranked results and keeps only the most relevant ones (typically the top 5).
    • This is like having an executive summary instead of a full report, ensuring the AI only works with the most important information.

    Citation Preservation

    • Keeps track of where each piece of information came from in your documents, including file names and page numbers.
    • Works like a reference manager in a Word document, automatically maintaining source information and allowing for proper attribution in responses.

    Visit original content creator repository
    https://github.com/manishkatyan/langgraph-chat-app

  • gnome-keyring-decryptor

    gnome-keyring-decryptor

    This repository contains a simple utility to decode and decrypt the .keystore-files
    used by the gnome-keyring utility and daemon.

    The decrypted keystore’s contents are then output in JSON format.

    I am really surprised this didn’t exist. There is a keyring-dump utility in the
    gnome-keyring-repo, but since the output-format of that utility is not really suitable
    for further processing, I decided to create this tool.

    Building & Hacking

    This project is built using meson, so you will need that.
    In order to use meson you will also need a build-backend. I simply use ninja
    which is the default.

    The provided nix-flake will yield a dev-environment containing all the necessary
    dependencies, so using it is recommended.

    Since this project also includes an .envrc using nix-direnv
    for use with the provided flake, I strongly recommend setting up direnv
    as well.

    Build depencies:

    • docopt.c: For generating the argument parser
      in accordance with the docopt-specification.

      Meson needs to be able to find this in your Python’s search-path.

    To build this project simply follow the usual meson-procedure:

    $ meson setup build
    $ ninja -C build

    Afterwards, if everything went well, the executable can be found under build/gnome-keyring-decryptor.

    Usage

    To use this tool simply call gnome-keyring-decryptor <keyring> with <keyring>
    being the path to the keyring-file on the command line.

    The contents of the keyring will then be output on stdout.

    References

    • The file format is documented in the gnome-keyring-repository under
      docs/file-format.txt.

      A copy of that file can be found in this repository – also under docs/file-format.txt
      in case the upstream version gets removed or moved in any way.

    • This tool is – in general – inspired by/copied from the aforementioned tool in the
      gnome-keyring-repo:
      dump-keyring0-format.c

    gnome-keyring in general and the dump-utility as well as the file format documentation
    specifically are licensed under the GNU General Public License Version 2.

    License

    This file is part of gnome-keyring-decryptor.

    gnome-keyring-decryptor is free software: you can redistribute it and/or modify it
    under the terms of the GNU General Public License as published by the Free Software Foundation,
    either version 3 of the License, or (at your option) any later version.

    gnome-keyring-decryptor is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
    without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    See the GNU General Public License for more details.

    You should have received a copy of the GNU General Public License along with gnome-keyring-decryptor.
    If not, see https://www.gnu.org/licenses/.

    Copyright 2023 Rufus Maurice Schäfing (wirklichniemand, nichtsundniemand)

    Visit original content creator repository
    https://github.com/nichtsundniemand/gnome-keyring-decryptor

  • hamt

    Hash Array Mapped Trie (HAMT)

    A HAMT is a hashmap/set stored as trie, which provides update and lookup performance similarly to a normal hashmap/set, but needs no rehashing and also allows one to copy the entire map/set in constant time. This implementation uses a 32 bit hash and trie nodes with 32 children, so 5 bits of the hash are consumed to choose the next child. When there are no hash collisions, this HAMT can store 2^32 items with a maximal tree depth of (log_32 2^32) = 6, i.e., you need 6 memory accesses to find any key/value which is practically O(1). Although when there are hash collisions, they are put in an array.

    Each HAMT node carries a reference counter, since FreePascal has no garbage collector. If the reference count is 1, the node can mutate, otherwise it is immutable with a copy-on-write semantic like strings. The counter is updated atomically, so the map could be shared across threads. This might lead to a large number of memory writes when a path of a full tree is copied (6 levels of up 32 children), but still less than copying a full hash table.

    Everything is implemented using generics, so it can be used with all types.

    Examples

    Mutable Map:

    type TMutableMapStringString = specialize TMutableMap<string, string, THAMTTypeInfo>;
    var map: TMutableMapStringString;
        p: TMutableMapStringString.PPair;
    begin
      map := TMutableMapStringString.create;
      map.Insert('hello', 'world');
      map.insert('foo', 'bar');
      map['abc'] := 'def';
    
      writeln(map['hello']); // world
      writeln(map.get('foo')); // bar
      writeln(map.get('abc', 'default')); // def
    
      //enumerate all
      for p in map do
        writeln(p^.key, ': ', p^.value);
    
      map.free;
    end.

    Immutable Map:

    type TImmutableMapStringString = specialize TImmutableMap<string, string, THAMTTypeInfo>;
    var map, map2, map3: TImmutableMapStringString;
        p: TImmutableMapStringString.PPair;
    begin
      map := TImmutableMapStringString.create;
      map2 := map.Insert('hello', 'world');
      map3 := map2.insert('foo', 'bar');
    
      writeln(map.get('hello', 'default')); // default
      writeln(map.get('foo', 'default')); // default
    
      writeln(map2.get('hello')); // world
      writeln(map2.get('foo', 'default')); // default
    
      writeln(map3['hello']); // world
      writeln(map3['foo']); // bar
    
      //enumerate all
      for p in map3 do
        writeln(p^.key, ': ', p^.value);
    
      map.free;
      map2.free;
      map3.free;
    end.

    Mutable Set:

    type TMutableSetString = specialize TMutableSet<string, THAMTTypeInfo>;
    var stringSet: TMutableSetString;
        p: TMutableSetString.PItem;
    begin
      stringSet := TMutableSetString.create;
      stringSet.Insert('hello');
      stringSet.insert('foo');
    
      writeln(stringSet['hello']); // true
      writeln(stringSet.contains('foo')); // true
      writeln(stringSet.contains('abc')); // false
    
      //enumerate all
      for p in stringSet do
        writeln(p^);
    
      stringSet.free;
    end.

    Immutable Set:

    type TImmutableSetString = specialize TImmutableSet<string, THAMTTypeInfo>;
    var set1, set2, set3: TImmutableSetString;
      p: TImmutableSetString.PItem;
    begin
      set1 := TImmutableSetString.create;
      set2 := set1.Insert('hello');
      set3 := set2.insert('foo');
    
      writeln(set1.contains('hello')); // false
      writeln(set1['foo']);            // false
    
      writeln(set2.contains('hello')); // true
      writeln(set2['foo']);            // false
    
      writeln(set3.contains('hello')); // true
      writeln(set3['foo']);            // true
    
      //enumerate all
      for p in set3 do
        writeln(p^);
    
      set1.free;
      set2.free;
      set3.free;
    end.

    Documentation

    Manual:

    Installation

    This library requires no installation, there are no dependencies besides the FreePascal compiler. Just copy the hamt.* files in the unit search path of FreePascal. Then you can use the maps with uses hamt.maps and the sets with uses hamt.sets.

    However, beware that the last stable release of FreePascal, 3.0.4, cannot compile the complex generics used here. You need to have at least FreePascal 3.1.1, preferably 3.3.1 with revision r39690.

    To run the tests cases in tests/hamt_tests.pas, you also need to have bbutils in the search path.

    References

    Ideal Hash Trees

    Efficient Immutable Collections

    Visit original content creator repository
    https://github.com/benibela/hamt

  • irk

    Apache License
    Version 2.0, January 2004
    http://www.apache.org/licenses/
    
    TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    
    1. Definitions.
    
    "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
    
    "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
    
    "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
    
    "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
    
    "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
    
    "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
    
    "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
    
    "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
    
    "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
    
    "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
    
    2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
    
    3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
    
    4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
    
         (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
    
         (b) You must cause any modified files to carry prominent notices stating that You changed the files; and
    
         (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
    
         (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
    
         You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
    
    5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
    
    6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
    
    7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
    
    8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
    
    9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
    
    END OF TERMS AND CONDITIONS
    
    APPENDIX: How to apply the Apache License to your work.
    
    To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!)  The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives.
    
    Copyright [yyyy] [name of copyright owner]
    
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at
    
    http://www.apache.org/licenses/LICENSE-2.0
    
    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
    

    Visit original content creator repository
    https://github.com/rniii/irk