Load huge amount of data in process 0 and use it in other processes with shared_memory

It happens that when we create multiprocesses say a node with 8XA100 GPUs that each process has a copy of data, which easily goes beyond the memory limit of the whole node. Instead, we could let the rank 0 load data and the rest wait until it is ready. The code is given as follows,

import time
import pickle
import numpy as np
from multiprocessing import shared_memory

# dummy data to load
data = [{"url": "xxx", "exp": "yyy"}] * 20

# serialize
data = [pickle.dumps(d) for d in data]

# process 0
if rank % 8 == 0:
	shm_a = shared_memory.ShareableList(data, name='shared_data')
    print(pickle.loads(shm_a[0]))
# other processes
else:
    while True:
        try:
            shm_b = shared_memory.ShareableList(name="shared_data")
            print(f"Attached to shared memory with name: shared_data")
            print(pickle.loads(shm_b[0]))
            break
        except FileNotFoundError:
            print("Shared memory not found. It may not be created yet.")
            time.sleep(100)

Is Lanzhou City the mass center of the map of China?

Almost!

It turns out that blue (Lanzhou city) is close to but not exact the mass center (red). I mean, if we only consider Lanzhou as a point.

An algorithmic approach

import json
import matplotlib.pyplot as plt
from matplotlib.patches import Polygon
import numpy as np
import cv2

# be aware to fix coordinates from 内蒙古 (add an extra bracket, download it from https://geo.datav.aliyun.com/areas_v3/bound/100000_full.json)
with open('100000_full.json', 'r') as f:
    content = json.load(f)

print(content.keys())


fig, ax = plt.subplots()

for feat in content['features']:
    if feat['properties']['name'] == '甘肃省':
        center = feat['properties']['center']
    if feat['properties']['name'] not in ['']:
        coordinates = feat['geometry']['coordinates']
        for c in coordinates:
            coord = np.array(c)
            if len(coord[0].shape) == 2:
                p = Polygon(coord[0], facecolor='k')
                ax.add_patch(p)
            else:
                print(coord[0])
                print(feat['properties']['name'])

ax.set_xlim([60, 140])
ax.set_ylim([0, 60])
cx, cy = center
print("center", center)
print("lim:", ax.get_xlim(), ax.get_ylim())
xmin, xmax = ax.get_xlim()
ymin, ymax = ax.get_ylim()
cx = (cx -60) * (640 / (xmax - xmin))
cy = 480 - cy * (480 / (ymax-ymin))
print("center scaled:", cx, cy)
plt.axis('off')
plt.tight_layout()
plt.savefig("china_map.png")
plt.show()


img = cv2.imread('china_map.png', 0)
print(img.shape, img.mean(), img.max(), img.min())
th, im_th = cv2.threshold(img, 100, 255, cv2.THRESH_BINARY_INV)

print(th)
print(im_th.shape)
cv2.imwrite('china_map_inverse.png', im_th)

moments = cv2.moments(im_th, True)
print(moments)

centeroid = (moments['m10'] / moments['m00'], moments['m01'] / moments['m00'])
print("centeroid:", centeroid)
x,y = centeroid
fig,ax = plt.subplots()

img = cv2.imread('china_map.png')
ax.imshow(img)

plt.scatter(x, y, c='r')
plt.scatter(cx, cy, c='b')
plt.tight_layout()
plt.axis('off')
plt.savefig('china_map_with_centeroid.png')
plt.show()

References

• https://blog.csdn.net/qq826309057/article/details/70039397
• https://www.zhoulujun.cn/html/GIS/WebGIS/8348.html
• http://datav.aliyun.com/portal/school/atlas/area_selector#&lat=33.50475906922609&lng=104.2822265625&zoom=4

Python: Find and Delete Corrupted Audio Files

import os
import wave
import glob

def compare_header_and_size(wav_filename):
    with wave.open(wav_filename, 'r') as fin:
            header_fsize = (fin.getnframes() * fin.getnchannels() * fin.getsampwidth()) + 44
    file_fsize = os.path.getsize(wav_filename)
    return header_fsize != file_fsize

print('The following files are corrupted:')
for f in  glob.glob("_background_noise_/*.wav"):
    if compare_header_and_size(f):
        print(f)
        os.remove(f)

Bootstrap Footer Best Practice

see Sticky Footer

html {
    position: relative;
    min-height: 100%;
}
body {
    margin-bottom: 60px;
}
.footer{
    position: absolute;
    bottom: 0;
    width: 100%;
    height: 60px;
    line-height: 60px;
    background-color: #f5f5f5;
}

Setting up Ubuntu 15.04 + PHP5.6 + Nginx + MySQL + Laravel 5 + npm + Bower + Gulp

1. install a brand new Ubuntu 15.04 amd64 version

2. uname -a

    Linux jasler 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

3. Setting root password

    sudo passwd root
    >> root

1. check if apache2 is installed and port 80 is being used

    ps -A | grep apache
    sudo netstat -nlp  | grep 80

remove if installed:

    sudo killall apache2
    sudo apt-get remove apache2

1. Install Nginx

    sudo apt-get install nginx
    nginx -v
        nginx version: nginx/1.6.2 (Ubuntu)

2. Install MySQL

    sudo apt-get install mysql-server

It will also install mysql-client and other dependencies in the meantime.

Press Enter on seeing a blank terminal to proceed.

Set the root password root for this time.

1. Install PHP5 and its extensions

    sudo apt-get install php5 php5-mcrypt php5-gd php5-mysql php5-fpm php5-curl

It also installs apache2 automatically, if it conflicts with nginx, we recommend to remove it.

Check the php and mysql version:

    php -v
    PHP 5.6.4-4ubuntu6.4 (cli) (built: Oct 28 2015 01:21:29) 
    Copyright (c) 1997-2014 The PHP Group
    Zend Engine v2.6.0, Copyright (c) 1998-2014 Zend Technologies
        with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2014, by Zend Technologies
 
 
    mysql -p -uroot -v
    Enter password: 
    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 4
    Server version: 5.6.28-0ubuntu0.15.04.1 (Ubuntu)

1. Install npm

   sudo apt-get install npm
   npm -v
   1.4.21

update within it self:

    sudo npm install -g npm
    /usr/local/bin/npm -> /usr/local/lib/node_modules/npm/bin/npm-cli.js
    npm@3.8.3 /usr/local/lib/node_modules/npm

Or install this way:

    sudo apt-get install curl
    curl -sL https://deb.nodesource.com/setup > install.sh
    sudo chmod +x install.sh
    sudo ./install.sh
    sudo apt-get install nodejs
    sudo npm install -g npm

1. Install git

   sudo apt-get install git

2. Creating dev folder:

change usr to your account

    sudo mkdir /home/wwwroot/
    sudo chown -R [usr].[usr] /home/wwwroot

Cloning repo:

cd /home/wwwroot/
git clone ssh://git@yourrepo.com:/home/git/yourrepo.git

npm install (within vpn)

sudo npm install

Install composer

cd ~
php -r "readfile('https://getcomposer.org/installer');" > composer-setup.php
php composer-setup.php
sudo mv composer.phar /etc/php5/composer.phar
sudo chmod +x /etc/php5/composer.phar
sudo ln -s /etc/php5/composer.phar /usr/bin/composer

Install dependencies (vpn):

cd /home/wwwroot/wwwroot
composer install

Create Database and user, privileges:

mysql -p -u root
 
> create database yourdb;
> create user 'you'@'localhost';
> set password for you@localhost = PASSWORD('yourpasswd');
> grant all privileges on yourdb.* to you@localhost;
> show grants for you@localhost;
> flush privileges;

reset config for environment

cp .env.example .env
 
vi .env
 
DB_HOST=127.0.0.1
DB_DATABASE=yourdb
DB_USERNAME=yourame
DB_PASSWORD=yourpasswd

Database migration

php artisan key:generate
    Application key [s1ZYF0BZydwbRRcrNc58crIC20uwBRzT] set successfully.
php artisan migrate
php artisan db:seed

Install bower, gulp

sudo npm install bower -g
sudo npm install gulp -g

Bower install dependencies (no-VPN prefered)

bower install

Generate css, js etc. frontend files

gulp
 
gulp watch

Make a directory for log files,

sudo mkdir /home/wwwlogs
sudo chown www-data.www-data /home/wwwlogs

1. Reload Nginx

   sudo nginx -s reload

Check the status:

    sudo service nginx status

1. Restart Nginx

   sudo service nginx restart

sudo mv yoursite.conf sites-available/

C: char* conversion to LPCWSTR

#include <string.h>
#include <strlib.h>

LPCWSTR c2ws(const char* s){
 int len;
 int slength = strlen(s)+1;
 len = MultiByteToWideChar(CP_ACP, 0, s, slength, 0, 0);
 wchar_t* buf = new wchar_t[len];
 MultiByteToWideChar(CP_ACP, 0, s, slength, buf, len);
 LPCWSTR r(buf);
 delete[] buf;
 return r;
}

Install Pygame for Python 2.7 on Windows 64bit

Check here for pygame wheel.

My config goes like C:\WinPython-64bit-2.7.10.3\python-2.7.10.amd64, so it is a pre-compiled python bundle. The official site lists release only for 32-bit. But it gives the link to a wheel link.

Basically, this is a personal managed wheel list. Choose the follow wheel and install through pip command

pip install pygame-1.9.2a0-cp27-none-win_amd64.whl

Refs

• http://www.lfd.uci.edu/~gohlke/pythonlibs/#pygame
• http://pygame.org/download.shtml