GolangNote

Golang笔记

用Golang 处理每分钟100万个请求

Permalink

用渠道作为线程池,应对百万请求,这是 Marcio Castilho 优化的方法。核心代码:

用Golang 处理每分钟100万个请求

Go: 用渠道作为线程池,应对百万请求
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
var (
 MaxWorker = os.Getenv("MAX_WORKERS")
 MaxQueue  = os.Getenv("MAX_QUEUE")
)

//Job represents the job to be run
type Job struct {
    Payload Payload
}

// A buffered channel that we can send work requests on.
var JobQueue chan Job

// Worker represents the worker that executes the job
type Worker struct {
    WorkerPool  chan chan Job
    JobChannel  chan Job
    quit        chan bool
}

func NewWorker(workerPool chan chan Job) Worker {
    return Worker{
        WorkerPool: workerPool,
        JobChannel: make(chan Job),
        quit:       make(chan bool)}
}

// Start method starts the run loop for the worker, listening for a quit channel in
// case we need to stop it
func (w Worker) Start() {
    go func() {
        for {
            // register the current worker into the worker queue.
            w.WorkerPool <- w.JobChannel

            select {
            case job := <-w.JobChannel:
                // we have received a work request.
                if err := job.Payload.UploadToS3(); err != nil {
                    log.Errorf("Error uploading to S3: %s", err.Error())
                }

            case <-w.quit:
                // we have received a signal to stop
                return
            }
        }
    }()
}

// Stop signals the worker to stop listening for work requests.
func (w Worker) Stop() {
    go func() {
        w.quit <- true
    }()
} 

//Handler
func payloadHandler(w http.ResponseWriter, r *http.Request) {

    if r.Method != "POST" {
		w.WriteHeader(http.StatusMethodNotAllowed)
		return
	}

    // Read the body into a string for json decoding
	var content = &PayloadCollection{}
	err := json.NewDecoder(io.LimitReader(r.Body, MaxLength)).Decode(&content)
    if err != nil {
		w.Header().Set("Content-Type", "application/json; charset=UTF-8")
		w.WriteHeader(http.StatusBadRequest)
		return
	}

    // Go through each payload and queue items individually to be posted to S3
    for _, payload := range content.Payloads {

        // let's create a job with the payload
        work := Job{Payload: payload}

        // Push the work onto the queue.
        JobQueue <- work
    }

    w.WriteHeader(http.StatusOK)
}

参见他的blog Handling 1 Million Requests per Minute with Golang

一个线程池库推荐 shettyh/threadpool

Related articles

Golang 单实例实现网站多域名请求

有时候写网站,为了统一的后端,把不同业务都集中到一个后端,这时就需要处理多域名的请求,在 Go http server 里实现很简单,只需把不同域名映射到不同的 `http.Handler`。...

Golang 把cookie 字符串解析为cookie 结构

在做爬虫时有时候会遇到需要带已登录的 cookie 请求,这个时候最简单的方法是在浏览器登录后,在开发者面板找到cookie 字符串,然后拷贝粘贴。这就面临一个问题需要把cookie 字符串解析成Go 语言 cookie 结构体。...

谷歌翻译的 golang 库推荐

Google 的翻译越来越好了,官方的Golang SDK 已经很完美,这里介绍的是几个非官方发布的有特色的库。...

Golang sync.WaitGroup 的 Wait 超时处理

sync.WaitGroup 使用 `Add(1)`、`Done()`、`Wait()`组合来实现多协程等待,如果某一协程未能合理处理错误,导致无法退出,此时需要引入超时机制。下面是一种超时处理方法。...

Write a Comment to "用Golang 处理每分钟100万个请求"

Submit Comment Login
Based on Golang + fastHTTP + sdb | go1.16.5 Processed in 1ms