Friday, September 6, 2024

Load huge amount of data in process 0 and use it in other processes with shared_memory

It happens that when we create multiprocesses say a node with 8XA100 GPUs that each process has a copy of data, which easily goes beyond the memory limit of the whole node. Instead, we could let the rank 0 load data and the rest wait until it is ready. The code is given as follows,

import time
import pickle
import numpy as np
from multiprocessing import shared_memory

# dummy data to load
data = [{"url": "xxx", "exp": "yyy"}] * 20

# serialize
data = [pickle.dumps(d) for d in data]

# process 0
if rank % 8 == 0:
	shm_a = shared_memory.ShareableList(data, name='shared_data')
    print(pickle.loads(shm_a[0]))
# other processes
else:
    while True:
        try:
            shm_b = shared_memory.ShareableList(name="shared_data")
            print(f"Attached to shared memory with name: shared_data")
            print(pickle.loads(shm_b[0]))
            break
        except FileNotFoundError:
            print("Shared memory not found. It may not be created yet.")
            time.sleep(100)

Saturday, December 3, 2022

Is Lanzhou City the mass center of the map of China?

Almost!

It turns out that blue (Lanzhou city) is close to but not exact the mass center (red). I mean, if we only consider Lanzhou as a point.

An algorithmic approach

import json
import matplotlib.pyplot as plt
from matplotlib.patches import Polygon
import numpy as np
import cv2

# be aware to fix coordinates from 内蒙古 (add an extra bracket, download it from https://geo.datav.aliyun.com/areas_v3/bound/100000_full.json)
with open('100000_full.json', 'r') as f:
    content = json.load(f)

print(content.keys())


fig, ax = plt.subplots()

for feat in content['features']:
    if feat['properties']['name'] == '甘肃省':
        center = feat['properties']['center']
    if feat['properties']['name'] not in ['']:
        coordinates = feat['geometry']['coordinates']
        for c in coordinates:
            coord = np.array(c)
            if len(coord[0].shape) == 2:
                p = Polygon(coord[0], facecolor='k')
                ax.add_patch(p)
            else:
                print(coord[0])
                print(feat['properties']['name'])

ax.set_xlim([60, 140])
ax.set_ylim([0, 60])
cx, cy = center
print("center", center)
print("lim:", ax.get_xlim(), ax.get_ylim())
xmin, xmax = ax.get_xlim()
ymin, ymax = ax.get_ylim()
cx = (cx -60) * (640 / (xmax - xmin))
cy = 480 - cy * (480 / (ymax-ymin))
print("center scaled:", cx, cy)
plt.axis('off')
plt.tight_layout()
plt.savefig("china_map.png")
plt.show()


img = cv2.imread('china_map.png', 0)
print(img.shape, img.mean(), img.max(), img.min())
th, im_th = cv2.threshold(img, 100, 255, cv2.THRESH_BINARY_INV)

print(th)
print(im_th.shape)
cv2.imwrite('china_map_inverse.png', im_th)

moments = cv2.moments(im_th, True)
print(moments)

centeroid = (moments['m10'] / moments['m00'], moments['m01'] / moments['m00'])
print("centeroid:", centeroid)
x,y = centeroid
fig,ax = plt.subplots()

img = cv2.imread('china_map.png')
ax.imshow(img)

plt.scatter(x, y, c='r')
plt.scatter(cx, cy, c='b')
plt.tight_layout()
plt.axis('off')
plt.savefig('china_map_with_centeroid.png')
plt.show()

References

  • https://blog.csdn.net/qq826309057/article/details/70039397
  • https://www.zhoulujun.cn/html/GIS/WebGIS/8348.html
  • http://datav.aliyun.com/portal/school/atlas/area_selector#&lat=33.50475906922609&lng=104.2822265625&zoom=4

Sunday, September 20, 2020

Python: Find and Delete Corrupted Audio Files

import os
import wave
import glob

def compare_header_and_size(wav_filename):
    with wave.open(wav_filename, 'r') as fin:
            header_fsize = (fin.getnframes() * fin.getnchannels() * fin.getsampwidth()) + 44
    file_fsize = os.path.getsize(wav_filename)
    return header_fsize != file_fsize

print('The following files are corrupted:')
for f in  glob.glob("_background_noise_/*.wav"):
    if compare_header_and_size(f):
        print(f)
        os.remove(f)

Sunday, August 20, 2017

Bootstrap Footer Best Practice

see Sticky Footer

html {
    position: relative;
    min-height: 100%;
}
body {
    margin-bottom: 60px;
}
.footer{
    position: absolute;
    bottom: 0;
    width: 100%;
    height: 60px;
    line-height: 60px;
    background-color: #f5f5f5;
}

Tuesday, June 7, 2016

Setting up Ubuntu 15.04 + PHP5.6 + Nginx + MySQL + Laravel 5 + npm + Bower + Gulp

  1. install a brand new Ubuntu 15.04 amd64 version
  2. uname -a

     Linux jasler 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
  3. Setting root password

     sudo passwd root
     >> root
  1. check if apache2 is installed and port 80 is being used

     ps -A | grep apache
     sudo netstat -nlp  | grep 80

remove if installed:

    sudo killall apache2
    sudo apt-get remove apache2
  1. Install Nginx

     sudo apt-get install nginx
     nginx -v
         nginx version: nginx/1.6.2 (Ubuntu)
  2. Install MySQL

     sudo apt-get install mysql-server

It will also install mysql-client and other dependencies in the meantime.

Press Enter on seeing a blank terminal to proceed.

Set the root password root for this time.

  1. Install PHP5 and its extensions

     sudo apt-get install php5 php5-mcrypt php5-gd php5-mysql php5-fpm php5-curl

It also installs apache2 automatically, if it conflicts with nginx, we recommend to remove it.

Check the php and mysql version:

    php -v
    PHP 5.6.4-4ubuntu6.4 (cli) (built: Oct 28 2015 01:21:29) 
    Copyright (c) 1997-2014 The PHP Group
    Zend Engine v2.6.0, Copyright (c) 1998-2014 Zend Technologies
        with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2014, by Zend Technologies
 
 
    mysql -p -uroot -v
    Enter password: 
    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 4
    Server version: 5.6.28-0ubuntu0.15.04.1 (Ubuntu)
  1. Install npm

    sudo apt-get install npm
    npm -v
    1.4.21

update within it self:

    sudo npm install -g npm
    /usr/local/bin/npm -> /usr/local/lib/node_modules/npm/bin/npm-cli.js
    npm@3.8.3 /usr/local/lib/node_modules/npm

Or install this way:

    sudo apt-get install curl
    curl -sL https://deb.nodesource.com/setup > install.sh
    sudo chmod +x install.sh
    sudo ./install.sh
    sudo apt-get install nodejs
    sudo npm install -g npm
  1. Install git

    sudo apt-get install git
  2. Creating dev folder:

change usr to your account

    sudo mkdir /home/wwwroot/
    sudo chown -R [usr].[usr] /home/wwwroot
  • Cloning repo:

    cd /home/wwwroot/
    git clone ssh://git@yourrepo.com:/home/git/yourrepo.git
  • npm install (within vpn)

    sudo npm install
  • Install composer

    cd ~
    php -r "readfile('https://getcomposer.org/installer');" > composer-setup.php
    php composer-setup.php
    sudo mv composer.phar /etc/php5/composer.phar
    sudo chmod +x /etc/php5/composer.phar
    sudo ln -s /etc/php5/composer.phar /usr/bin/composer
  • Install dependencies (vpn):

    cd /home/wwwroot/wwwroot
    composer install
  • Create Database and user, privileges:

    mysql -p -u root
     
    > create database yourdb;
    > create user 'you'@'localhost';
    > set password for you@localhost = PASSWORD('yourpasswd');
    > grant all privileges on yourdb.* to you@localhost;
    > show grants for you@localhost;
    > flush privileges;
  • reset config for environment

    cp .env.example .env
     
    vi .env
     
    DB_HOST=127.0.0.1
    DB_DATABASE=yourdb
    DB_USERNAME=yourame
    DB_PASSWORD=yourpasswd
  • Database migration

    php artisan key:generate
        Application key [s1ZYF0BZydwbRRcrNc58crIC20uwBRzT] set successfully.
    php artisan migrate
    php artisan db:seed
  • Install bower, gulp

    sudo npm install bower -g
    sudo npm install gulp -g
  • Bower install dependencies (no-VPN prefered)

    bower install
  • Generate css, js etc. frontend files

    gulp
     
    gulp watch
  • Make a directory for log files,

    sudo mkdir /home/wwwlogs
    sudo chown www-data.www-data /home/wwwlogs
    1. Reload Nginx

      sudo nginx -s reload

    Check the status:

        sudo service nginx status
    1. Restart Nginx

      sudo service nginx restart

    sudo mv yoursite.conf sites-available/

    C: char* conversion to LPCWSTR

    #include <string.h>
    #include <strlib.h>
    
    LPCWSTR c2ws(const char* s){
     int len;
     int slength = strlen(s)+1;
     len = MultiByteToWideChar(CP_ACP, 0, s, slength, 0, 0);
     wchar_t* buf = new wchar_t[len];
     MultiByteToWideChar(CP_ACP, 0, s, slength, buf, len);
     LPCWSTR r(buf);
     delete[] buf;
     return r;
    }
    

    Friday, May 20, 2016

    Install Pygame for Python 2.7 on Windows 64bit

    Check here for pygame wheel.

    My config goes like C:\WinPython-64bit-2.7.10.3\python-2.7.10.amd64, so it is a pre-compiled python bundle. The official site lists release only for 32-bit. But it gives the link to a wheel link.

    Basically, this is a personal managed wheel list. Choose the follow wheel and install through pip command

    pip install pygame-1.9.2a0-cp27-none-win_amd64.whl

    Refs