论坛徽章:: 0

电梯直达

1楼 [收藏(0)] [报告]

发表于 2013-01-30 15:06 |只看该作者 |倒序浏览

本帖最后由 tyguaike 于 2013-01-30 16:48 编辑

怎么写一个脚本能从百度音乐top 500里下载音乐到我的电脑里

http://music.baidu.com/top/dayhot

百度Top 500下载经过分析可以实现，
但是本人的Shell水平有限，现把实现的思路整理出来。

第一步，取得Top 500的页面
SOURCE="music.baidu.com/top/dayhot"
wget ${SOURCE} -O top500

提取歌曲名和 id
cat top500 | grep "'sid'.*'sname'.*'author'"

得到下面的行怎么用正则得到 "391713 海阔天空" 生成一个文件列表
  data-songitem = '{"songItem":{"sid":"391713","author":"\u4fe1\u4e50\u56e2"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '391713', 'sname': '海阔天空', 'author': '>信乐团' } } ">

wget "music.baidu.com/song/"${id}"/download?_o=%2Ftop%2Fdayhot" -O tmp

这里得到一个含有下载连接的页面 tmp
cat tmp | grep "<a.*xcode"
                  <a href="/data/music/file?link=http://zhangmenshiting.baidu.com/data2/music/33971202/23473715212400128.mp3?xcode=e2c42face0c3558110a0c31806938d9e " id="download" hidefocus="true" class="btn-download">下载</a><span class="btn-download-span"></span>
又要从这个页面里用正则得到这一行地址：
http://zhangmenshiting.baidu.com ... 58110a0c31806938d9e
然后就可以用这一行地址下载到歌典了

*********************************下面是比较完整的数据***************************************8
提取歌曲名和 id
cat top500 | grep "'sid'.*'sname'.*'author'"
   <li  data-songitem = '{"songItem":{"sid":"23473715","author":&quot

sy"}}' class="lead-top first bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '23473715', 'sname': '江南Style', 'author': 'Psy' } } ">
   <li  data-songitem = '{"songItem":{"sid":"14950804","author":"\u66f2\u5a49\u5a77"}}' class="lead-top  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '14950804', 'sname': '我的歌声里', 'author': '曲婉婷' } } ">
   <li  data-songitem = '{"songItem":{"sid":"5966719","author":"\u51e4\u51f0\u4f20\u5947"}}' class="lead-top  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '5966719', 'sname': '最炫民族风', 'author': '凤凰传奇' } } ">
   <li  data-songitem = '{"songItem":{"sid":"1575873","author":"\u6c6a\u5cf0"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '1575873', 'sname': '北京北京', 'author': '汪峰' } } ">
   <li  data-songitem = '{"songItem":{"sid":"207737","author":"\u8fea\u514b\u725b\u4ed4"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '207737', 'sname': '放手去爱', 'author': '迪克牛仔' } } ">
   <li  data-songitem = '{"songItem":{"sid":"2104023","author":"\u4f83\u4f83"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '2104023', 'sname': '滴答', 'author': '侃侃' } } ">
   <li  data-songitem = '{"songItem":{"sid":"12380958","author":"\u5d14\u5b50\u683c"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '12380958', 'sname': '老婆最大', 'author': '崔子格' } } ">
   <li  data-songitem = '{"songItem":{"sid":"5071402","author":"\u9ad8\u8fdb,\u5c0f\u6c88\u9633"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '5071402', 'sname': '我的好兄弟', 'author': '高进,小沈阳' } } ">
   <li  data-songitem = '{"songItem":{"sid":"536010","author":"\u51e4\u51f0\u4f20\u5947"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '536010', 'sname': '荷塘月色', 'author': '凤凰传奇' } } ">
   <li  data-songitem = '{"songItem":{"sid":"7313936","author":"\u738b\u83f2,\u9648\u5955\u8fc5"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '7313936', 'sname': '因为爱情', 'author': '王菲,陈奕迅' } } ">
   <li  data-songitem = '{"songItem":{"sid":"13139228","author":"\u5f20\u60e0\u59b9"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '13139228', 'sname': '哭砂', 'author': '张惠妹' } } ">
   <li  data-songitem = '{"songItem":{"sid":"24492803","author":"\u5e73\u5b89"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '24492803', 'sname': '洋葱', 'author': '平安' } } ">
   <li  data-songitem = '{"songItem":{"sid":"2488100","author":"\u5218\u82e5\u82f1"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '2488100', 'sname': '为爱痴狂', 'author': '刘若英' } } ">
   <li  data-songitem = '{"songItem":{"sid":"5837361","author":"\u738b\u9e9f"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '5837361', 'sname': '伤不起', 'author': '王麟' } } ">
   <li  data-songitem = '{"songItem":{"sid":"223901","author":"\u674e\u4e3d\u82ac"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '223901', 'sname': '爱江山更爱美人', 'author': '李丽芬' } } ">
   <li  data-songitem = '{"songItem":{"sid":"2126014","author":"\u90d1\u667a\u5316"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '2126014', 'sname': '水手', 'author': '郑智化' } } ">
   <li  data-songitem = '{"songItem":{"sid":"14880387","author":"\u738b\u9e9f"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '14880387', 'sname': '思密达', 'author': '王麟' } } ">
   <li  data-songitem = '{"songItem":{"sid":"13762260","author":"\u9ad8\u5b89,\u676d\u5a07"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '13762260', 'sname': '一生无悔', 'author': '高安,杭娇' } } ">
   <li  data-songitem = '{"songItem":{"sid":"2121739","author":"\u6768\u94b0\u83b9"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '2121739', 'sname': '轻轻的告诉你', 'author': '杨钰莹' } } ">
   <li  data-songitem = '{"songItem":{"sid":"31496563","author":"\u5468\u6770\u4f26"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '31496563', 'sname': '红尘客栈', 'author': '周杰伦' } } ">
   <li  data-songitem = '{"songItem":{"sid":"974156","author":"\u5c60\u6d2a\u521a"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '974156', 'sname': '精忠报国', 'author': '屠洪刚' } } ">
   <li  data-songitem = '{"songItem":{"sid":"28790755","author":"\u5c0f\u6c88\u9633,\u6c88\u6625\u9633"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '28790755', 'sname': '你懂得', 'author': '小沈阳,沈春阳' } } ">
   <li  data-songitem = '{"songItem":{"sid":"10509017","author":"\u51b7\u6f20,\u6768\u5c0f\u66fc"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '10509017', 'sname': '我爱你胜过你爱我', 'author': '冷漠,杨小曼' } } ">
   <li  data-songitem = '{"songItem":{"sid":"10554229","author":"\u6c6a\u5cf0"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '10554229', 'sname': '存在', 'author': '汪峰' } } ">
   <li  data-songitem = '{"songItem":{"sid":"23149293","author":"\u674e\u4ee3\u6cab"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '23149293', 'sname': '我的歌声里', 'author': '李代沫' } } ">
   <li  data-songitem = '{"songItem":{"sid":"416940","author":"\u90d1\u6e90"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '416940', 'sname': '包容', 'author': '郑源' } } ">
   <li  data-songitem = '{"songItem":{"sid":"1026041","author":"\u5f20\u9707\u5cb3"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '1026041', 'sname': '爱我别走', 'author': '张震岳' } } ">
   <li  data-songitem = '{"songItem":{"sid":"23305140","author":"\u51e4\u51f0\u4f20\u5947"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '23305140', 'sname': '中国味道', 'author': '凤凰传奇' } } ">
   <li  data-songitem = '{"songItem":{"sid":"7316463","author":"\u9648\u5955\u8fc5"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '7316463', 'sname': '爱情转移', 'author': '陈奕迅' } } ">
   <li  data-songitem = '{"songItem":{"sid":"7333126","author":"\u9093\u4e3d\u541b"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '7333126', 'sname': '我只在乎你', 'author': '邓丽君' } } ">
   <li  data-songitem = '{"songItem":{"sid":"23159920","author":"\u90d1\u8679,\u674e\u4ee3\u6cab"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '23159920', 'sname': '如果没有你', 'author': '郑虹,李代沫' } } ">
   <li  data-songitem = '{"songItem":{"sid":"1449893","author":"\u6885\u8273\u82b3"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '1449893', 'sname': '亲密爱人', 'author': '梅艳芳' } } ">
   <li  data-songitem = '{"songItem":{"sid":"31387633","author":"\u5409\u514b\u96bd\u9038"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '31387633', 'sname': '彩色的黑', 'author': '吉克隽逸' } } ">
   <li  data-songitem = '{"songItem":{"sid":"27131218","author":"\u5409\u514b\u96bd\u9038"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '27131218', 'sname': 'Halo', 'author': '吉克隽逸' } } ">
   <li  data-songitem = '{"songItem":{"sid":"233076","author":"\u6c6a\u5cf0"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '233076', 'sname': '怒放的生命', 'author': '汪峰' } } ">
   <li  data-songitem = '{"songItem":{"sid":"5835631","author":"\u5c0f\u6c88\u9633"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '5835631', 'sname': '大笑江湖', 'author': '小沈阳' } } ">
   <li  data-songitem = '{"songItem":{"sid":"13139680","author":"\u5f20\u60e0\u59b9"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '13139680', 'sname': '我可以抱你吗', 'author': '张惠妹' } } ">
   <li  data-songitem = '{"songItem":{"sid":"416996","author":"\u90d1\u6e90"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '416996', 'sname': '爱情码头', 'author': '郑源' } } ">
   <li  data-songitem = '{"songItem":{"sid":"23150239","author":"\u591a\u4eae"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '23150239', 'sname': '小情歌', 'author': '多亮' } } ">
   <li  data-songitem = '{"songItem":{"sid":"274606","author":"\u5f20\u5b66\u53cb"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '274606', 'sname': '心如刀割', 'author': '张学友' } } ">
   <li  data-songitem = '{"songItem":{"sid":"7288694","author":"\u90a3\u82f1"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '7288694', 'sname': '相见不如怀念', 'author': '那英' } } ">
   <li  data-songitem = '{"songItem":{"sid":"247937","author":"\u96f7\u8bfa\u513f"}}' class="  bb-dotimg clearfix  song-item-hook { 'songItem': { 'sid': '247937', 'sname': '别在我离开之前离开', 'author': '雷诺儿' } } ">

文库|博客

reyleon

大富大贵

论坛徽章:: 60

2楼 [报告]

发表于 2013-01-30 15:32 |只看该作者

我也想知道。。。

实战分享：从技术角度谈机器学习入门| 【大话IT】RadonDB低门槛向MySQL集群下战书 | ChinaUnix打赏功能已上线！ | 新一代分布式关系型数据库RadonDB知多少？

reyleon

大富大贵

论坛徽章:: 60

3楼 [报告]

发表于 2013-01-30 15:42 |只看该作者

回复 1# tyguaike

貌似可以实现喔，等我研究下

实战分享：从技术角度谈机器学习入门| 【大话IT】RadonDB低门槛向MySQL集群下战书 | ChinaUnix打赏功能已上线！ | 新一代分布式关系型数据库RadonDB知多少？

刺客阿地

小富即安

论坛徽章:: 11

4楼 [报告]

发表于 2013-01-30 15:44 |只看该作者

回复 3# reyleon

坐等脚本出现，哈哈！

实战分享：从技术角度谈机器学习入门| 【大话IT】RadonDB低门槛向MySQL集群下战书 | ChinaUnix打赏功能已上线！ | 新一代分布式关系型数据库RadonDB知多少？

ziyunfei

稍有积蓄

论坛徽章:: 1

5楼 [报告]

发表于 2013-01-30 15:53 |只看该作者

很简单嘛都在页面里

实战分享：从技术角度谈机器学习入门| 【大话IT】RadonDB低门槛向MySQL集群下战书 | ChinaUnix打赏功能已上线！ | 新一代分布式关系型数据库RadonDB知多少？

tyguaike

稍有积蓄

论坛徽章:: 0

6楼 [报告]

发表于 2013-01-30 17:20 |只看该作者

怎么没人帮个忙呀，就是一些正则

实战分享：从技术角度谈机器学习入门| 【大话IT】RadonDB低门槛向MySQL集群下战书 | ChinaUnix打赏功能已上线！ | 新一代分布式关系型数据库RadonDB知多少？

ziyunfei

稍有积蓄

论坛徽章:: 1

7楼 [报告]

发表于 2013-01-30 17:35 |只看该作者

一种方法是从下载页入手,一行命令,一些管道组成.

wget -O- http://music.baidu.com/top/dayhot | grep -oP '(?<=.sid.: .)\d+' | sed 's#.*#http://music.baidu.com/song/&/download/#' | wget -O- -i- | grep -oP '(?<=song_title:")[^"]+|(?<=singer_name:")[^"]+|(?<=<a href="/data/music/file\?link=)[^"]+|(?<=<a href=")http.+?(?= " id="download")|height:70px' | awk 'NR%3==1{url=$0}NR%3==2{name=$0}NR%3==0{system("wget -O \""name".mp3\" \""url"\"")}'

复制代码

不幸的是,下载页格式不统一.有三种:
http://music.baidu.com/song/1449893/download?__o=%2Ftop%2Fdayhot
http://music.baidu.com/song/922519/download?__o=%2Fsong%2F922519
http://music.baidu.com/song/23473715/download?__o=%2Ftop%2Fdayhot

会导致一些歌曲无法下载.

另外一种比较稳定简单的方法是通过播放页的请求歌曲url的ajax接口 http://play.baidu.com/data/music/songlink?songIds=1449893
有个问题是,返回的中文名是Unicode编码的,比如"songName":"\u4eb2\u5bc6\u7231\u4eba",你要用一些工具(比如iconv,甚至python,perl)将\u4eb2\u5bc6\u7231\u4eba转换成汉字"亲密爱人".

第二种方式,我没写,不过稍微改一改,就行了.

实战分享：从技术角度谈机器学习入门| 【大话IT】RadonDB低门槛向MySQL集群下战书 | ChinaUnix打赏功能已上线！ | 新一代分布式关系型数据库RadonDB知多少？

ljwd1000

丰衣足食

论坛徽章:: 0

8楼 [报告]

发表于 2013-01-30 17:40 |只看该作者

import os,sys,re,urllib
#coding=utf-8
try:
from progressbar import *
except:
print "Please run easy_install progressbar!"
sys.exit(1)
url = "http://music.baidu.com/top/new"
BasePath = os.path.dirname(os.path.abspath(__file__))
BasePath = BasePath + "/test"
def download_bar():
class CrazyFileTransferSpeed(FileTransferSpeed):
"""It's bigger between 45 and 80 percent."""
def update(self, pbar):
if 45 < pbar.percentage() < 80:
return 'Bigger Now ' + FileTransferSpeed.update(self,pbar)
else:
return FileTransferSpeed.update(self,pbar)
widgets = [CrazyFileTransferSpeed(),' <<<', Bar(), '>>> ',
Percentage(),' ', ETA()]
pbar = ProgressBar(widgets=widgets, maxval=10000000)
# maybe do something
pbar.start()
for i in range(2000000):
# do something
pbar.update(5*i+1)
pbar.finish()
if not os.path.exists(BasePath):
os.makedirs(BasePath)
fd = urllib.urlopen(url)
content = fd.read()
pattern = '<li.*sid\':\ \'(\d+)\'.*sname\':\ \'([^\']*)\''
matchs = re.findall(pattern,content,re.M)
for url in matchs:
x = url[1].decode('utf-8')
url = "http://music.baidu.com/song/%s/download" %(url[0])
fd_d = urllib.urlopen(url)
content_d = fd_d.read()
pattern_d = '<.*downlink.*(http://zhangmenshiting.baidu.com[^"]*).*'
matchs_d = re.findall(pattern_d,content_d,re.M)
for url_d in matchs_d:
if url_d:
filename = "%s.mp3" %(x)
print "%s is start download" % (filename)
output = "%s/%s" % (BasePath, filename)
urllib.urlretrieve(url_d, output, reporthook=download_bar())
print "%s is finished download" %(filename)
print "\n\n"
fd_d.close()
fd.close()

复制代码

回复 1# tyguaike

这个是之前用python写的，你可以参考一下

实战分享：从技术角度谈机器学习入门| 【大话IT】RadonDB低门槛向MySQL集群下战书 | ChinaUnix打赏功能已上线！ | 新一代分布式关系型数据库RadonDB知多少？

reyleon

大富大贵

论坛徽章:: 60

9楼 [报告]

发表于 2013-01-30 17:58 |只看该作者

本帖最后由 reyleon 于 2013-01-30 20:20 编辑

高品质与超高品质要登录才能下载，不知道怎么通过命令登录，故下载不了高品质与超高品质的音乐；所以下面的脚本默认没登录只能下载标准品质的，姑且看看吧！！！另外，求高手指教如何使用curl方式登录。

#!/bin/bash

curl -s "http://music.baidu.com/top/dayhot"|awk 'BEGIN{FS="\"";OFS="\t"}/\/song\/[0-9]+/{print $2,"\""$4"\""}' > music_menu.txt

Count=0
Thread=15
FifoFile="$$.fifo"
mkfifo $FifoFile
exec 6<>$FifoFile
rm $FifoFile
for ((i=0;i<=$Thread;i++));do echo;done >&6
exec 5<music_menu.txt
while read -u5 line
do
      read -u6
      let Count+=1
      eval $(echo "$line"|awk 'BEGIN{FS="\"|\t"}{print "musicurl="$1,"music="gensub(/ /,"_","g",$(NF-1))}')
      baseurl='http://music.baidu.com'
      fullurl="$baseurl/$musicurl/download?__o=%2Ftop%2Fdayhot"
      echo -e "正在下载第 $Count 首,${music}.mp3"
      {
            downloadurl=$(curl -s "$fullurl"|grep '/data/music/file'|awk 'BEGIN{FS="\""}/下载/{print $2}')
            wget -q -O ${music}.mp3 "$baseurl/$downloadurl"
            echo >&6
            sleep 1
      } &
done

wait
exec 6>&-
exec 5>&-

实战分享：从技术角度谈机器学习入门| 【大话IT】RadonDB低门槛向MySQL集群下战书 | ChinaUnix打赏功能已上线！ | 新一代分布式关系型数据库RadonDB知多少？

tyguaike

稍有积蓄

论坛徽章:: 0

10楼 [报告]

发表于 2013-01-30 18:39 |只看该作者

回复 9# reyleon

高手哈，SHELL这么强大哈。
不过有个问题

ongdong@ubuntu:~/temp/aa$ sh gg
gg: 11: gg: Syntax error: Bad for loop variable

实战分享：从技术角度谈机器学习入门| 【大话IT】RadonDB低门槛向MySQL集群下战书 | ChinaUnix打赏功能已上线！ | 新一代分布式关系型数据库RadonDB知多少？

12 3 / 3 页下一页

返回列表

Chinaunix › 论坛 › 程序设计 › Shell › 批量下载音乐

[网络相关] 批量下载音乐 [复制链接]