Lexin Gong

Creating Custom Stickers/Avatars from Your Webcam - Part 1

2024-03-04T00:00:00+00:00

I’ve just made my new project, Poker Face, available as open-source. This project is crafted with Elixir and Phoenix LiveView. Its core features allow you to:

Transform your webcam feed into stickers or avatars.
Pose questions about the camera images to GPT.

Check out the introduction video here: In this blog series, I’ll guide you through every detail of the implementation. Welcome to Part 1.

In this very first part, we’ll tackle a couple of key steps:

Opening the camera and capturing the video frame.
Sending the frame to a LiveView process and interfacing it with Gemini or OpenAI.

Initial Project

Let’s kick things off with setting up the project. Navigate to your desired directory and create a new phoenix project:

mix phx.new poker_face

From the outset, my goal was to build this entire project in LiveView. I proceeded by adding face_live.ex and face_live.html.heex, then configuring the router to point to this LiveView component.

defmodule PokerFaceWeb.FaceLive do
  use PokerFaceWeb, :live_view

  @impl true
  def mount(_prams, _session, socket) do
    {:ok, assign(socket, :photo_info, nil)}
  end
end

<div>
  <.header>
    Poker Face
    <:subtitle></:subtitle>
  </.header>
  <div class="grid grid-cols-2 gap-x-2">
    <div id="user_photo" class="rounded shadow-lg p-2 flex flex-col gap-y-2" phx-hook="Camera">
      <button
        id="startCamera"
        class="p-4 w-full h-full disabled:cursor-not-allowed"
      >
        <p class="select-none flex items-center gap-2">
          <.icon name="hero-camera" />Take a photo with your webcam
        </p>
      </button>
      <div class="flex flex-col gap-y-2">
        <video id="video" class="hidden" playsinline autoplay></video>
        <img id="photo" />

        <div id="buttonGroup" class="hidden flex flex-col gap-y-2 justify-center">
          <button
            id="takePhoto"
            class="rounded-full py-2 border border-black hover:bg-black hover:text-white"
          >
            Take Photo
          </button>
          <button
            id="stopCamera"
            class="rounded-full py-2 border border-black hover:bg-black hover:text-white"
          >
            Cancel
          </button>
        </div>
        <canvas id="canvas" class="hidden"></canvas>
      </div>
    </div>
    <div id="poker_photo" class="rounded shadow-lg p-2">
      <p class="text-center text-lg"><%= @photo_info %></p>
    </div>
  </div>
</div>

defmodule PokerFaceWeb.Router do
  use PokerFaceWeb, :router

  ...

  scope "/", PokerFaceWeb do
    pipe_through :browser

    live "/", FaceLive
  end

  ...

Once you run mix phx.server, your page should like this:

The variable @photo_info is designated to display the responses from GPT/Gemini once we’ve sent an video frame to be analyzed.

Webcam

Now, let’s delve into the webcam functionality. To be honest, webcams were somewhat foreign to me. So, I turned to GPT for assistance on how to implement it, and it provided a solid, functional code example. All I had to do was integrate the JavaScript into a Phoenix Hook.

Add this code to app.js with any necessary modifications:

let Hooks = {}

Hooks.Camera = {
  mounted() {
    const startCamera = document.getElementById('startCamera');
    const video = document.getElementById('video');
    const takePhoto = document.getElementById('takePhoto');
    const stopCamera = document.getElementById('stopCamera');
    const canvas = document.getElementById('canvas');
    const buttonGroup = document.getElementById('buttonGroup');
    const photo = document.getElementById("photo");
    let stream = null;

    // Start the camera when button is clicked
    startCamera.addEventListener('click', async () => {
      stream = await navigator.mediaDevices.getUserMedia({ video: true });
      video.srcObject = stream;
      video.classList.remove('hidden');
      buttonGroup.classList.remove('hidden');
      startCamera.classList.add('hidden');
      photo.src = "";
    });

    // Take a photo
    takePhoto.addEventListener('click', () => {
      canvas.width = video.videoWidth;
      canvas.height = video.videoHeight;
      canvas.getContext('2d').drawImage(video, 0, 0);
      const imageDataURL = canvas.toDataURL('image/png');
      photo.src = imageDataURL;

      // push to the back-end and interact with Gemini/GPT
      this.pushEvent("new_photo", { photo: imageDataURL })

      // buttonGroup.classList.add('hidden');
      // startCamera.classList.remove('hidden');
      // video.classList.add('hidden');
      // video.srcObject = null;
    });

    // Stop the camera
    stopCamera.addEventListener('click', () => {
      if (stream) {
        stream.getTracks().forEach(track => track.stop());
      }
      buttonGroup.classList.add('hidden');
      startCamera.classList.remove('hidden');
      video.classList.add('hidden');
      video.srcObject = null;
      photo.src = "";
    });
  }
}

By clicking the camera button, the camera will activate: Cool, it works. GPT is great!

Click the ‘Take Photo’ button though, and you’ll bump into an error. Currently, when this button is clicked, an event named new_photo is dispatched to the backend, for which the handling is not yet implemented. Let’s proceed with that.

LiveView process

Edit face_live.ex and add these functions:

...

  @impl true
  def handle_event("new_photo", %{"photo" => photo}, socket) do
    send(self(), {:anaylse, photo})
    {:noreply, socket}
  end

  @impl true
  def handle_info({:anaylse, photo}, socket) do
    {:ok, text} = Gemini.analyze_image(photo)

    socket =
      socket
      |> assign(:photo_info, text)

    {:noreply, socket}
  end

...

Instead of OpenAI gpt, We implement Gemini API first, Let’s add a gemini.ex:

defmodule PokerFace.Gemini do
  @vision_uri "https://generativelanguage.googleapis.com/v1beta/models/gemini-pro-vision:generateContent"

  def analyze_image("data:image/png;base64," <> image_data) do
    prompt = """
      What's is he/she/it doing?
    """

    body = %{
      contents: [
        %{
          parts: [
            %{text: prompt},
            %{inlineData: %{mimeType: "image/png", data: image_data}}
          ]
        }
      ]
    }

    resp =
      Req.post!(@vision_uri <> "?key=#{System.get_env("GOOGLE_AI_API_KEY")}",
        json: body,
        receive_timeout: 60_000,
        connect_options: [protocols: [:http1]]
      )

    text =
      resp.body["candidates"]
      |> List.first()
      |> Map.get("content")
      |> Map.get("parts")
      |> List.first()
      |> Map.get("text")

    {:ok, text}
  end
end

In order to utilize Gemini, you must obtain your personal API key from Google AI. Once acquired and configured as an environment variable, you’ll possess a functional AI-powered image analyzer.

Although Gemini responds correctly, the photo captured becomes non-persistent on the current page. So, we need to make some fix and an OpenAI version could also be integrated. I’ll not show the code here. But you can check all the code changes We should make today.

At the end of these modifications, we are equipped with a fully operational camera AI image analyzer!

Managing Elixir's main branch with asdf

2024-01-10T00:00:00+00:00

I’ve always used asdf to manage Elixir versions, and I’ve also had a persistent need to run projects or experiments with the latest code from the Elixir repo (main branch). Installing the main branch is quite simple:

Clone the Elixir code and build it locally:

git clone https://github.com/elixir-lang/elixir.git /path/to/elixir
cd /path/to/elixir && make

Create a symlink in the asdf installs directory and switch to the main version:

ln -s /path/to/elixir ~/.asdf/installs/elixir/main
asdf reshim elixir main
asdf shell elixir main | asdf global elixir main

Verify:

$ asdf list elixir
  1.15.7-otp-26
  1.16.0-otp-26
 *main

Daily update:
```
cd /path/to/elixir && git pull && make
```

You can write the above update commands to a script and run it periodically to ensure you’re always using the latest version of the main branch.
If running make directly throws compilation errors, you need to delete the previous build artifacts before rebuilding:
```
cd /path/to/elixir
make clean && make
```
Using the main branch can be somewhat unstable as it’s still under development.

在安卓手机上安装 Elixir

2019-07-25T00:00:00+00:00

记录一下在安卓手机上安装 Elixir 的过程。

在手机上安装 Termux 应用
开启 SSH 服务（非必须, 开 ssh 只是为了便于 PC 端访问操作手机，比在手机上打字方便）
- 安装相关包
```
  apt update
  apt upgrade
  pkg install openssh
  pkg install net-tools
```
- 设置密码并开启 ssh
```
  # 设置一个密码
  passwd
  # 开启服务
  sshd
```
- 查看手机 ip
```
  ifconfig
```

安装 Elixir

从 PC 端登录手机

  # 确保手机和 PC 处在同一网络，用上一步获取的安卓手机 ip 登录
  # 拷贝公钥到安卓手机(需输入前面设置的密码)
  ssh-copy-id -p 8022 -i your_public_key_file andriod_ip
  # 登录
  ssh -p 8022 android_ip

安装相关包 (PC 端登录之后，就可以开始正式的安装步骤了)
```
  pkg install unzip
  pkg install erlang
```

安装 Elixir, 参考 https://github.com/hexpm/bob#elixir-builds 确定好 elixir 和 erlang otp 版本, 这里以 elixir 1.9.1 otp 22 为例:

  mkdir elixir
  cd elixir
  # wget https://repo.hex.pm/builds/elixir/{REF}-otp-{OTP_MAJOR_VERSION}.zip
  wget https://repo.hex.pm/builds/elixir/v1.9.1-otp-22.zip
  unzip v1.9.1-otp-22.zip
  rm v1.9.1-otp-22.zip

设置 PATH, 保存到 bashrc

  vi .bashrc
  export PATH=$PATH:$HOME/elixir/bin

验证

elixir -v
#> Erlang/OTP 22 [erts-10.4.4] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1]
#>
#> Elixir 1.9.1 (compiled with Erlang/OTP 22)

Mac 下临时显示隐藏的文件和文件夹

2013-12-21T00:00:00+00:00

今天遇到一个问题, 在使用 Github 的 Mac 客户端的时候, 想直接导入已经 clone 到本地的 repo, 但是这个 repo 目录是一个以 . 为开头的格式命名的隐藏目录, 在打开的文件选择窗口里面是无法看到的。

以前我一般是打开终端输入如下命令:

defaults write com.apple.finder AppleShowAllFiles TRUE; killall Finder

这条命令有一个不便利的地方就是执行之后系统就会一直显示那些隐藏文件，往往等你操作完之后，你又需要把值重新设置回去:

defaults write com.apple.finder AppleShowAllFiles FALSE; killall Finder

今天才发现，原来在选择文件的时候, 只需要按下 Command + Shift + . 组合键就可以临时地显示隐藏文件了。当然这不仅仅适用于这一个软件，Mac 下所有软件的文件选择窗口都是可以这么做的。

参考: http://www.tekrevue.com/tip/how-to-temporarily-see-hidden-files-folders-in-mac-os-x/

Rails 使用 mongoDB 和 Slim

2012-11-24T00:00:00+00:00

预言中的世界末日都要来了，期待的 Ruby 2.0 和 Rails 4.0 都还没发布。那我就来看看怎么集成 mongoDB 和 Slim 到一个 Rails 工程。

安装 mongoDB 数据库

brew install mongodb 安装好之后，可以把 mongo 的服务设为开机启动，或者在需要的时候手动 mongod 开启。

新建 Rails 工程

rails new projectname --skip-test-unit --skip-active-record --skip-bundle

因为使用 mongoDB 并打算使用 minitest 所以会在创建工程的时候 skip 掉不必要的东西。

修改 Gemfile

在 Gemfile 里面添加以下几句之后执行 bundle install

gem 'slim-rails'
gem 'mongoid'
group :test, :development do
  gem 'minitest-rails'
end

生成数据库配置和 minitest helper 文件

rails generate mongoid:config
rails generate mini_test:install

Slim 页面

把 application.html.erb 转换成 slim 格式，可以使用 html2slim 或者自己手动修改。参见我改好的版本

使用

现在就可以使用 rails g model | controller 等等命令了，默认生成的 view 文件都会是 slim 格式，model 都会自动 include mongoid 的 class。enjoy it!