免费注册 查看新帖 |

Chinaunix

  平台 论坛 博客 文库
最近访问板块 发新帖
查看: 31451 | 回复: 5

Delightful Database Seeding with Docker [复制链接]

论坛徽章:
32
CU大牛徽章
日期:2013-05-20 10:45:13每日论坛发贴之星
日期:2015-09-07 06:20:00每日论坛发贴之星
日期:2015-09-07 06:20:00数据库技术版块每日发帖之星
日期:2015-12-13 06:20:0015-16赛季CBA联赛之江苏
日期:2016-03-03 11:56:13IT运维版块每日发帖之星
日期:2016-03-06 06:20:00fulanqi
日期:2016-06-17 17:54:25IT运维版块每日发帖之星
日期:2016-07-23 06:20:0015-16赛季CBA联赛之佛山
日期:2016-08-11 18:06:41JAVA
日期:2016-10-25 16:09:072017金鸡报晓
日期:2017-01-10 15:13:292017金鸡报晓
日期:2017-02-08 10:33:21
发表于 2015-09-20 15:19 |显示全部楼层
Our entire application stack is packaged using Docker. One particular aspect of getting our application stack up and running is the initial data population. Having streamlined this, we’re able to have a complete environment up and running within minutes.

Prior to setting up seeding with Docker we often had problems with outdated examples and leftover data from previous demos. We also spent too much time manually keeping everything in sync. As our customer base grew, this was not a sustainable situation.

Using Docker we have created a user friendly tool that anyone in our organization can use, (even sales) both when resetting a demo laptop, and when getting a new on-site installation quickly up and running.
docker1.png

论坛徽章:
32
CU大牛徽章
日期:2013-05-20 10:45:13每日论坛发贴之星
日期:2015-09-07 06:20:00每日论坛发贴之星
日期:2015-09-07 06:20:00数据库技术版块每日发帖之星
日期:2015-12-13 06:20:0015-16赛季CBA联赛之江苏
日期:2016-03-03 11:56:13IT运维版块每日发帖之星
日期:2016-03-06 06:20:00fulanqi
日期:2016-06-17 17:54:25IT运维版块每日发帖之星
日期:2016-07-23 06:20:0015-16赛季CBA联赛之佛山
日期:2016-08-11 18:06:41JAVA
日期:2016-10-25 16:09:072017金鸡报晓
日期:2017-01-10 15:13:292017金鸡报晓
日期:2017-02-08 10:33:21
发表于 2015-09-20 15:19 |显示全部楼层
Now, let’s dive into the details!

We maintain all our demo workspaces, tutorials and help texts in our SaaS-production environment – this makes it easier to keep everything up-to-date. When running offline demos or setting up on-site solutions at customers, we need a way to copy data from the production environment to seed theses installations.

Prior to using Docker this was an error prone and ad-hoc activity. After moving to Docker we are now able to do this as part of every build. The process of preparing and applying a data seed includes the following steps:

Copy production data and clean it
Package and distribute the seed data
Import the data

Step 1 – Cleaning the database dump
To filter out other environment specific meta-data, we need to clean up the database dump before it is distributed. The core of the clean up is a throw away MongoDB Docker container. The reason we have the separate step with the throw away container is a security precaution – we don’t want to expose filtered out data in a lower Docker file system layer.

We start the container with the database dump available in a mounted volume, fire up MongoDB, run a clean up script in MongoDB, and finally dump the cleaned database to the mounted volume. The Docker container is discarded, so the only side effect of this process is the cleaned database dump.
docker2.png

论坛徽章:
32
CU大牛徽章
日期:2013-05-20 10:45:13每日论坛发贴之星
日期:2015-09-07 06:20:00每日论坛发贴之星
日期:2015-09-07 06:20:00数据库技术版块每日发帖之星
日期:2015-12-13 06:20:0015-16赛季CBA联赛之江苏
日期:2016-03-03 11:56:13IT运维版块每日发帖之星
日期:2016-03-06 06:20:00fulanqi
日期:2016-06-17 17:54:25IT运维版块每日发帖之星
日期:2016-07-23 06:20:0015-16赛季CBA联赛之佛山
日期:2016-08-11 18:06:41JAVA
日期:2016-10-25 16:09:072017金鸡报晓
日期:2017-01-10 15:13:292017金鸡报晓
日期:2017-02-08 10:33:21
发表于 2015-09-20 15:20 |显示全部楼层
Step 2 – Packaging the data for distribution
For easy distribution, we package the database dump, the binary files, the MongoDB dump, and restore tools in one Docker image.

Dockerfile: FROM mongo:3.0.1 ADD cleaned.tar.gz /work ADD attachments.tar.gz /work

docker3.png

论坛徽章:
32
CU大牛徽章
日期:2013-05-20 10:45:13每日论坛发贴之星
日期:2015-09-07 06:20:00每日论坛发贴之星
日期:2015-09-07 06:20:00数据库技术版块每日发帖之星
日期:2015-12-13 06:20:0015-16赛季CBA联赛之江苏
日期:2016-03-03 11:56:13IT运维版块每日发帖之星
日期:2016-03-06 06:20:00fulanqi
日期:2016-06-17 17:54:25IT运维版块每日发帖之星
日期:2016-07-23 06:20:0015-16赛季CBA联赛之佛山
日期:2016-08-11 18:06:41JAVA
日期:2016-10-25 16:09:072017金鸡报晓
日期:2017-01-10 15:13:292017金鸡报晓
日期:2017-02-08 10:33:21
发表于 2015-09-20 15:20 |显示全部楼层
Step 3 – Distribution
Once the data is packaged along with the MongoDB client, we upload the image to our private account on Docker Hub. Right now this is triggered manually, automation is just a matter of configuration.

This allows our developers, sales people and customers to update their local installations by downloading the latest version by pulling the latest image from Docker Hub.
docker4.png

论坛徽章:
32
CU大牛徽章
日期:2013-05-20 10:45:13每日论坛发贴之星
日期:2015-09-07 06:20:00每日论坛发贴之星
日期:2015-09-07 06:20:00数据库技术版块每日发帖之星
日期:2015-12-13 06:20:0015-16赛季CBA联赛之江苏
日期:2016-03-03 11:56:13IT运维版块每日发帖之星
日期:2016-03-06 06:20:00fulanqi
日期:2016-06-17 17:54:25IT运维版块每日发帖之星
日期:2016-07-23 06:20:0015-16赛季CBA联赛之佛山
日期:2016-08-11 18:06:41JAVA
日期:2016-10-25 16:09:072017金鸡报晓
日期:2017-01-10 15:13:292017金鸡报晓
日期:2017-02-08 10:33:21
发表于 2015-09-20 15:21 |显示全部楼层
Step 4 – Seeding
Since all data is packaged along with the MongoDB client, seeding is a breeze. It is basically a matter of running two docker commands:

To populate the database, the seed image is started with a link to the MongoDB container, and restores the database using the bundled database dump:

docker run --rm --link ardoq_mongodb_1:mongodb ardoq/demo-seed mongorestore -h mongodb /work/demo_seed/

To populate the binary attachments, the seed image is started again with the data image volumes mounted, and simply copies the files over and exits:

docker run --rm --volumes-from ardoq_api_1 ardoq/demo-seed:latest cp -r /work/attachments /data

docker5.png

论坛徽章:
32
CU大牛徽章
日期:2013-05-20 10:45:13每日论坛发贴之星
日期:2015-09-07 06:20:00每日论坛发贴之星
日期:2015-09-07 06:20:00数据库技术版块每日发帖之星
日期:2015-12-13 06:20:0015-16赛季CBA联赛之江苏
日期:2016-03-03 11:56:13IT运维版块每日发帖之星
日期:2016-03-06 06:20:00fulanqi
日期:2016-06-17 17:54:25IT运维版块每日发帖之星
日期:2016-07-23 06:20:0015-16赛季CBA联赛之佛山
日期:2016-08-11 18:06:41JAVA
日期:2016-10-25 16:09:072017金鸡报晓
日期:2017-01-10 15:13:292017金鸡报晓
日期:2017-02-08 10:33:21
发表于 2015-09-20 15:21 |显示全部楼层
Summary

Keeping on-site installations up-to-date with the latest data used to be time consuming and error prone. Using Docker we are well on our way to automating the entire process.

您需要登录后才可以回帖 登录 | 注册

本版积分规则 发表回复

  

北京盛拓优讯信息技术有限公司. 版权所有 京ICP备16024965号-6 北京市公安局海淀分局网监中心备案编号:11010802020122 niuxiaotong@pcpop.com 17352615567
未成年举报专区
中国互联网协会会员  联系我们:huangweiwei@itpub.net
感谢所有关心和支持过ChinaUnix的朋友们 转载本站内容请注明原作者名及出处

清除 Cookies - ChinaUnix - Archiver - WAP - TOP