Java队列和线程池消费处理的简单设计

分手后的思念是犯贱 2022-12-26 04:46 194阅读 0赞

**目录**

实现目标

目标分析

线程池设计

队列与线程池结合

--------------------

# 实现目标  #

*  队列缓冲业务数据
 *  线程空闲忙碌识别
 *  任务处理的进度控制

# 目标分析 #

进度控制：需要将队列里面总任务和线程消费的任务进行记录，实现一个completed/total  这样的控制。

线程池控制：线程池使用计数器，在完成任务和新增任务进行原子操作维护计数器数量。

# 线程池设计 #

线程池实际上就是一个线程的池化处理，一般会初始化几个线程，需要使用线程时从池子里面拿，池子里面的总容量占用多少可以用来标记线程繁忙和空闲。线程池的设计跟数据库JDBC的设计非常相似。比如获取连接多久超时等等，在线程池里面也是可以这样来实现，线程的示例是用来执行任务的，线程池大小的个数不宜太大，一般core\*2或者core\*2+1 ，再或者凑个十进制十位整数。

**队列Queue：**队列可以是一个链表，也可以是一个简单的集合，需要设计相应的队列排队策略（入队、出队、优先级），还需要设计相应的方法来便于外部操作，Java队列主要有以下操作方法：

![70][]

**线程池：**线程池的实现也有很多，比如最常用的Excutors的多种类型线程池，比如：

*  自动伸缩类型线程池
 *  固定大小类型线程池
 *  单个线程池
 *  调度式线程池
 *  不可配置线程池...

package java.util.concurrent;
 import java.util.*;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.security.AccessControlContext;
 import java.security.AccessController;
 import java.security.PrivilegedAction;
 import java.security.PrivilegedExceptionAction;
 import java.security.PrivilegedActionException;
 import java.security.AccessControlException;
 import sun.security.util.SecurityConstants;
 
 /**
 * Factory and utility methods for {@link Executor}, {@link
 * ExecutorService}, {@link ScheduledExecutorService}, {@link
 * ThreadFactory}, and {@link Callable} classes defined in this
 * package. This class supports the following kinds of methods:
 *
 * <ul>
 * <li> Methods that create and return an {@link ExecutorService}
 * set up with commonly useful configuration settings.
 * <li> Methods that create and return a {@link ScheduledExecutorService}
 * set up with commonly useful configuration settings.
 * <li> Methods that create and return a "wrapped" ExecutorService, that
 * disables reconfiguration by making implementation-specific methods
 * inaccessible.
 * <li> Methods that create and return a {@link ThreadFactory}
 * that sets newly created threads to a known state.
 * <li> Methods that create and return a {@link Callable}
 * out of other closure-like forms, so they can be used
 * in execution methods requiring {@code Callable}.
 * </ul>
 *
 * @since 1.5
 * @author Doug Lea
 */
 public class Executors {
 
 /**
 * Creates a thread pool that reuses a fixed number of threads
 * operating off a shared unbounded queue. At any point, at most
 * {@code nThreads} threads will be active processing tasks.
 * If additional tasks are submitted when all threads are active,
 * they will wait in the queue until a thread is available.
 * If any thread terminates due to a failure during execution
 * prior to shutdown, a new one will take its place if needed to
 * execute subsequent tasks. The threads in the pool will exist
 * until it is explicitly {@link ExecutorService#shutdown shutdown}.
 *
 * @param nThreads the number of threads in the pool
 * @return the newly created thread pool
 * @throws IllegalArgumentException if {@code nThreads <= 0}
 */
 public static ExecutorService newFixedThreadPool(int nThreads) {
 return new ThreadPoolExecutor(nThreads, nThreads,
 0L, TimeUnit.MILLISECONDS,
 new LinkedBlockingQueue<Runnable>());
 }
 
 /**
 * Creates a thread pool that maintains enough threads to support
 * the given parallelism level, and may use multiple queues to
 * reduce contention. The parallelism level corresponds to the
 * maximum number of threads actively engaged in, or available to
 * engage in, task processing. The actual number of threads may
 * grow and shrink dynamically. A work-stealing pool makes no
 * guarantees about the order in which submitted tasks are
 * executed.
 *
 * @param parallelism the targeted parallelism level
 * @return the newly created thread pool
 * @throws IllegalArgumentException if {@code parallelism <= 0}
 * @since 1.8
 */
 public static ExecutorService newWorkStealingPool(int parallelism) {
 return new ForkJoinPool
 (parallelism,
 ForkJoinPool.defaultForkJoinWorkerThreadFactory,
 null, true);
 }
 
 /**
 * Creates a work-stealing thread pool using all
 * {@link Runtime#availableProcessors available processors}
 * as its target parallelism level.
 * @return the newly created thread pool
 * @see #newWorkStealingPool(int)
 * @since 1.8
 */
 public static ExecutorService newWorkStealingPool() {
 return new ForkJoinPool
 (Runtime.getRuntime().availableProcessors(),
 ForkJoinPool.defaultForkJoinWorkerThreadFactory,
 null, true);
 }
 
 /**
 * Creates a thread pool that reuses a fixed number of threads
 * operating off a shared unbounded queue, using the provided
 * ThreadFactory to create new threads when needed. At any point,
 * at most {@code nThreads} threads will be active processing
 * tasks. If additional tasks are submitted when all threads are
 * active, they will wait in the queue until a thread is
 * available. If any thread terminates due to a failure during
 * execution prior to shutdown, a new one will take its place if
 * needed to execute subsequent tasks. The threads in the pool will
 * exist until it is explicitly {@link ExecutorService#shutdown
 * shutdown}.
 *
 * @param nThreads the number of threads in the pool
 * @param threadFactory the factory to use when creating new threads
 * @return the newly created thread pool
 * @throws NullPointerException if threadFactory is null
 * @throws IllegalArgumentException if {@code nThreads <= 0}
 */
 public static ExecutorService newFixedThreadPool(int nThreads, ThreadFactory threadFactory) {
 return new ThreadPoolExecutor(nThreads, nThreads,
 0L, TimeUnit.MILLISECONDS,
 new LinkedBlockingQueue<Runnable>(),
 threadFactory);
 }
 
 /**
 * Creates an Executor that uses a single worker thread operating
 * off an unbounded queue. (Note however that if this single
 * thread terminates due to a failure during execution prior to
 * shutdown, a new one will take its place if needed to execute
 * subsequent tasks.) Tasks are guaranteed to execute
 * sequentially, and no more than one task will be active at any
 * given time. Unlike the otherwise equivalent
 * {@code newFixedThreadPool(1)} the returned executor is
 * guaranteed not to be reconfigurable to use additional threads.
 *
 * @return the newly created single-threaded Executor
 */
 public static ExecutorService newSingleThreadExecutor() {
 return new FinalizableDelegatedExecutorService
 (new ThreadPoolExecutor(1, 1,
 0L, TimeUnit.MILLISECONDS,
 new LinkedBlockingQueue<Runnable>()));
 }
 
 /**
 * Creates an Executor that uses a single worker thread operating
 * off an unbounded queue, and uses the provided ThreadFactory to
 * create a new thread when needed. Unlike the otherwise
 * equivalent {@code newFixedThreadPool(1, threadFactory)} the
 * returned executor is guaranteed not to be reconfigurable to use
 * additional threads.
 *
 * @param threadFactory the factory to use when creating new
 * threads
 *
 * @return the newly created single-threaded Executor
 * @throws NullPointerException if threadFactory is null
 */
 public static ExecutorService newSingleThreadExecutor(ThreadFactory threadFactory) {
 return new FinalizableDelegatedExecutorService
 (new ThreadPoolExecutor(1, 1,
 0L, TimeUnit.MILLISECONDS,
 new LinkedBlockingQueue<Runnable>(),
 threadFactory));
 }
 
 /**
 * Creates a thread pool that creates new threads as needed, but
 * will reuse previously constructed threads when they are
 * available. These pools will typically improve the performance
 * of programs that execute many short-lived asynchronous tasks.
 * Calls to {@code execute} will reuse previously constructed
 * threads if available. If no existing thread is available, a new
 * thread will be created and added to the pool. Threads that have
 * not been used for sixty seconds are terminated and removed from
 * the cache. Thus, a pool that remains idle for long enough will
 * not consume any resources. Note that pools with similar
 * properties but different details (for example, timeout parameters)
 * may be created using {@link ThreadPoolExecutor} constructors.
 *
 * @return the newly created thread pool
 */
 public static ExecutorService newCachedThreadPool() {
 return new ThreadPoolExecutor(0, Integer.MAX_VALUE,
 60L, TimeUnit.SECONDS,
 new SynchronousQueue<Runnable>());
 }
 
 /**
 * Creates a thread pool that creates new threads as needed, but
 * will reuse previously constructed threads when they are
 * available, and uses the provided
 * ThreadFactory to create new threads when needed.
 * @param threadFactory the factory to use when creating new threads
 * @return the newly created thread pool
 * @throws NullPointerException if threadFactory is null
 */
 public static ExecutorService newCachedThreadPool(ThreadFactory threadFactory) {
 return new ThreadPoolExecutor(0, Integer.MAX_VALUE,
 60L, TimeUnit.SECONDS,
 new SynchronousQueue<Runnable>(),
 threadFactory);
 }
 
 /**
 * Creates a single-threaded executor that can schedule commands
 * to run after a given delay, or to execute periodically.
 * (Note however that if this single
 * thread terminates due to a failure during execution prior to
 * shutdown, a new one will take its place if needed to execute
 * subsequent tasks.) Tasks are guaranteed to execute
 * sequentially, and no more than one task will be active at any
 * given time. Unlike the otherwise equivalent
 * {@code newScheduledThreadPool(1)} the returned executor is
 * guaranteed not to be reconfigurable to use additional threads.
 * @return the newly created scheduled executor
 */
 public static ScheduledExecutorService newSingleThreadScheduledExecutor() {
 return new DelegatedScheduledExecutorService
 (new ScheduledThreadPoolExecutor(1));
 }
 
 /**
 * Creates a single-threaded executor that can schedule commands
 * to run after a given delay, or to execute periodically. (Note
 * however that if this single thread terminates due to a failure
 * during execution prior to shutdown, a new one will take its
 * place if needed to execute subsequent tasks.) Tasks are
 * guaranteed to execute sequentially, and no more than one task
 * will be active at any given time. Unlike the otherwise
 * equivalent {@code newScheduledThreadPool(1, threadFactory)}
 * the returned executor is guaranteed not to be reconfigurable to
 * use additional threads.
 * @param threadFactory the factory to use when creating new
 * threads
 * @return a newly created scheduled executor
 * @throws NullPointerException if threadFactory is null
 */
 public static ScheduledExecutorService newSingleThreadScheduledExecutor(ThreadFactory threadFactory) {
 return new DelegatedScheduledExecutorService
 (new ScheduledThreadPoolExecutor(1, threadFactory));
 }
 
 /**
 * Creates a thread pool that can schedule commands to run after a
 * given delay, or to execute periodically.
 * @param corePoolSize the number of threads to keep in the pool,
 * even if they are idle
 * @return a newly created scheduled thread pool
 * @throws IllegalArgumentException if {@code corePoolSize < 0}
 */
 public static ScheduledExecutorService newScheduledThreadPool(int corePoolSize) {
 return new ScheduledThreadPoolExecutor(corePoolSize);
 }
 
 /**
 * Creates a thread pool that can schedule commands to run after a
 * given delay, or to execute periodically.
 * @param corePoolSize the number of threads to keep in the pool,
 * even if they are idle
 * @param threadFactory the factory to use when the executor
 * creates a new thread
 * @return a newly created scheduled thread pool
 * @throws IllegalArgumentException if {@code corePoolSize < 0}
 * @throws NullPointerException if threadFactory is null
 */
 public static ScheduledExecutorService newScheduledThreadPool(
 int corePoolSize, ThreadFactory threadFactory) {
 return new ScheduledThreadPoolExecutor(corePoolSize, threadFactory);
 }
 
 /**
 * Returns an object that delegates all defined {@link
 * ExecutorService} methods to the given executor, but not any
 * other methods that might otherwise be accessible using
 * casts. This provides a way to safely "freeze" configuration and
 * disallow tuning of a given concrete implementation.
 * @param executor the underlying implementation
 * @return an {@code ExecutorService} instance
 * @throws NullPointerException if executor null
 */
 public static ExecutorService unconfigurableExecutorService(ExecutorService executor) {
 if (executor == null)
 throw new NullPointerException();
 return new DelegatedExecutorService(executor);
 }
 
 /**
 * Returns an object that delegates all defined {@link
 * ScheduledExecutorService} methods to the given executor, but
 * not any other methods that might otherwise be accessible using
 * casts. This provides a way to safely "freeze" configuration and
 * disallow tuning of a given concrete implementation.
 * @param executor the underlying implementation
 * @return a {@code ScheduledExecutorService} instance
 * @throws NullPointerException if executor null
 */
 public static ScheduledExecutorService unconfigurableScheduledExecutorService(ScheduledExecutorService executor) {
 if (executor == null)
 throw new NullPointerException();
 return new DelegatedScheduledExecutorService(executor);
 }
 
 /**
 * Returns a default thread factory used to create new threads.
 * This factory creates all new threads used by an Executor in the
 * same {@link ThreadGroup}. If there is a {@link
 * java.lang.SecurityManager}, it uses the group of {@link
 * System#getSecurityManager}, else the group of the thread
 * invoking this {@code defaultThreadFactory} method. Each new
 * thread is created as a non-daemon thread with priority set to
 * the smaller of {@code Thread.NORM_PRIORITY} and the maximum
 * priority permitted in the thread group. New threads have names
 * accessible via {@link Thread#getName} of
 * pool-N-thread-M, where N is the sequence
 * number of this factory, and M is the sequence number
 * of the thread created by this factory.
 * @return a thread factory
 */
 public static ThreadFactory defaultThreadFactory() {
 return new DefaultThreadFactory();
 }
 
 /**
 * Returns a thread factory used to create new threads that
 * have the same permissions as the current thread.
 * This factory creates threads with the same settings as {@link
 * Executors#defaultThreadFactory}, additionally setting the
 * AccessControlContext and contextClassLoader of new threads to
 * be the same as the thread invoking this
 * {@code privilegedThreadFactory} method. A new
 * {@code privilegedThreadFactory} can be created within an
 * {@link AccessController#doPrivileged AccessController.doPrivileged}
 * action setting the current thread's access control context to
 * create threads with the selected permission settings holding
 * within that action.
 *
 * Note that while tasks running within such threads will have
 * the same access control and class loader settings as the
 * current thread, they need not have the same {@link
 * java.lang.ThreadLocal} or {@link
 * java.lang.InheritableThreadLocal} values. If necessary,
 * particular values of thread locals can be set or reset before
 * any task runs in {@link ThreadPoolExecutor} subclasses using
 * {@link ThreadPoolExecutor#beforeExecute(Thread, Runnable)}.
 * Also, if it is necessary to initialize worker threads to have
 * the same InheritableThreadLocal settings as some other
 * designated thread, you can create a custom ThreadFactory in
 * which that thread waits for and services requests to create
 * others that will inherit its values.
 *
 * @return a thread factory
 * @throws AccessControlException if the current access control
 * context does not have permission to both get and set context
 * class loader
 */
 public static ThreadFactory privilegedThreadFactory() {
 return new PrivilegedThreadFactory();
 }
 
 /**
 * Returns a {@link Callable} object that, when
 * called, runs the given task and returns the given result. This
 * can be useful when applying methods requiring a
 * {@code Callable} to an otherwise resultless action.
 * @param task the task to run
 * @param result the result to return
 * @param <T> the type of the result
 * @return a callable object
 * @throws NullPointerException if task null
 */
 public static <T> Callable<T> callable(Runnable task, T result) {
 if (task == null)
 throw new NullPointerException();
 return new RunnableAdapter<T>(task, result);
 }
 
 /**
 * Returns a {@link Callable} object that, when
 * called, runs the given task and returns {@code null}.
 * @param task the task to run
 * @return a callable object
 * @throws NullPointerException if task null
 */
 public static Callable<Object> callable(Runnable task) {
 if (task == null)
 throw new NullPointerException();
 return new RunnableAdapter<Object>(task, null);
 }
 
 /**
 * Returns a {@link Callable} object that, when
 * called, runs the given privileged action and returns its result.
 * @param action the privileged action to run
 * @return a callable object
 * @throws NullPointerException if action null
 */
 public static Callable<Object> callable(final PrivilegedAction<?> action) {
 if (action == null)
 throw new NullPointerException();
 return new Callable<Object>() {
 public Object call() { return action.run(); }};
 }
 
 /**
 * Returns a {@link Callable} object that, when
 * called, runs the given privileged exception action and returns
 * its result.
 * @param action the privileged exception action to run
 * @return a callable object
 * @throws NullPointerException if action null
 */
 public static Callable<Object> callable(final PrivilegedExceptionAction<?> action) {
 if (action == null)
 throw new NullPointerException();
 return new Callable<Object>() {
 public Object call() throws Exception { return action.run(); }};
 }
 
 /**
 * Returns a {@link Callable} object that will, when called,
 * execute the given {@code callable} under the current access
 * control context. This method should normally be invoked within
 * an {@link AccessController#doPrivileged AccessController.doPrivileged}
 * action to create callables that will, if possible, execute
 * under the selected permission settings holding within that
 * action; or if not possible, throw an associated {@link
 * AccessControlException}.
 * @param callable the underlying task
 * @param <T> the type of the callable's result
 * @return a callable object
 * @throws NullPointerException if callable null
 */
 public static <T> Callable<T> privilegedCallable(Callable<T> callable) {
 if (callable == null)
 throw new NullPointerException();
 return new PrivilegedCallable<T>(callable);
 }
 
 /**
 * Returns a {@link Callable} object that will, when called,
 * execute the given {@code callable} under the current access
 * control context, with the current context class loader as the
 * context class loader. This method should normally be invoked
 * within an
 * {@link AccessController#doPrivileged AccessController.doPrivileged}
 * action to create callables that will, if possible, execute
 * under the selected permission settings holding within that
 * action; or if not possible, throw an associated {@link
 * AccessControlException}.
 *
 * @param callable the underlying task
 * @param <T> the type of the callable's result
 * @return a callable object
 * @throws NullPointerException if callable null
 * @throws AccessControlException if the current access control
 * context does not have permission to both set and get context
 * class loader
 */
 public static <T> Callable<T> privilegedCallableUsingCurrentClassLoader(Callable<T> callable) {
 if (callable == null)
 throw new NullPointerException();
 return new PrivilegedCallableUsingCurrentClassLoader<T>(callable);
 }
 
 // Non-public classes supporting the public methods
 
 /**
 * A callable that runs given task and returns given result
 */
 static final class RunnableAdapter<T> implements Callable<T> {
 final Runnable task;
 final T result;
 RunnableAdapter(Runnable task, T result) {
 this.task = task;
 this.result = result;
 }
 public T call() {
 task.run();
 return result;
 }
 }
 
 /**
 * A callable that runs under established access control settings
 */
 static final class PrivilegedCallable<T> implements Callable<T> {
 private final Callable<T> task;
 private final AccessControlContext acc;
 
 PrivilegedCallable(Callable<T> task) {
 this.task = task;
 this.acc = AccessController.getContext();
 }
 
 public T call() throws Exception {
 try {
 return AccessController.doPrivileged(
 new PrivilegedExceptionAction<T>() {
 public T run() throws Exception {
 return task.call();
 }
 }, acc);
 } catch (PrivilegedActionException e) {
 throw e.getException();
 }
 }
 }
 
 /**
 * A callable that runs under established access control settings and
 * current ClassLoader
 */
 static final class PrivilegedCallableUsingCurrentClassLoader<T> implements Callable<T> {
 private final Callable<T> task;
 private final AccessControlContext acc;
 private final ClassLoader ccl;
 
 PrivilegedCallableUsingCurrentClassLoader(Callable<T> task) {
 SecurityManager sm = System.getSecurityManager();
 if (sm != null) {
 // Calls to getContextClassLoader from this class
 // never trigger a security check, but we check
 // whether our callers have this permission anyways.
 sm.checkPermission(SecurityConstants.GET_CLASSLOADER_PERMISSION);
 
 // Whether setContextClassLoader turns out to be necessary
 // or not, we fail fast if permission is not available.
 sm.checkPermission(new RuntimePermission("setContextClassLoader"));
 }
 this.task = task;
 this.acc = AccessController.getContext();
 this.ccl = Thread.currentThread().getContextClassLoader();
 }
 
 public T call() throws Exception {
 try {
 return AccessController.doPrivileged(
 new PrivilegedExceptionAction<T>() {
 public T run() throws Exception {
 Thread t = Thread.currentThread();
 ClassLoader cl = t.getContextClassLoader();
 if (ccl == cl) {
 return task.call();
 } else {
 t.setContextClassLoader(ccl);
 try {
 return task.call();
 } finally {
 t.setContextClassLoader(cl);
 }
 }
 }
 }, acc);
 } catch (PrivilegedActionException e) {
 throw e.getException();
 }
 }
 }
 
 /**
 * The default thread factory
 */
 static class DefaultThreadFactory implements ThreadFactory {
 private static final AtomicInteger poolNumber = new AtomicInteger(1);
 private final ThreadGroup group;
 private final AtomicInteger threadNumber = new AtomicInteger(1);
 private final String namePrefix;
 
 DefaultThreadFactory() {
 SecurityManager s = System.getSecurityManager();
 group = (s != null) ? s.getThreadGroup() :
 Thread.currentThread().getThreadGroup();
 namePrefix = "pool-" +
 poolNumber.getAndIncrement() +
 "-thread-";
 }
 
 public Thread newThread(Runnable r) {
 Thread t = new Thread(group, r,
 namePrefix + threadNumber.getAndIncrement(),
 0);
 if (t.isDaemon())
 t.setDaemon(false);
 if (t.getPriority() != Thread.NORM_PRIORITY)
 t.setPriority(Thread.NORM_PRIORITY);
 return t;
 }
 }
 
 /**
 * Thread factory capturing access control context and class loader
 */
 static class PrivilegedThreadFactory extends DefaultThreadFactory {
 private final AccessControlContext acc;
 private final ClassLoader ccl;
 
 PrivilegedThreadFactory() {
 super();
 SecurityManager sm = System.getSecurityManager();
 if (sm != null) {
 // Calls to getContextClassLoader from this class
 // never trigger a security check, but we check
 // whether our callers have this permission anyways.
 sm.checkPermission(SecurityConstants.GET_CLASSLOADER_PERMISSION);
 
 // Fail fast
 sm.checkPermission(new RuntimePermission("setContextClassLoader"));
 }
 this.acc = AccessController.getContext();
 this.ccl = Thread.currentThread().getContextClassLoader();
 }
 
 public Thread newThread(final Runnable r) {
 return super.newThread(new Runnable() {
 public void run() {
 AccessController.doPrivileged(new PrivilegedAction<Void>() {
 public Void run() {
 Thread.currentThread().setContextClassLoader(ccl);
 r.run();
 return null;
 }
 }, acc);
 }
 });
 }
 }
 
 /**
 * A wrapper class that exposes only the ExecutorService methods
 * of an ExecutorService implementation.
 */
 static class DelegatedExecutorService extends AbstractExecutorService {
 private final ExecutorService e;
 DelegatedExecutorService(ExecutorService executor) { e = executor; }
 public void execute(Runnable command) { e.execute(command); }
 public void shutdown() { e.shutdown(); }
 public List<Runnable> shutdownNow() { return e.shutdownNow(); }
 public boolean isShutdown() { return e.isShutdown(); }
 public boolean isTerminated() { return e.isTerminated(); }
 public boolean awaitTermination(long timeout, TimeUnit unit)
 throws InterruptedException {
 return e.awaitTermination(timeout, unit);
 }
 public Future<?> submit(Runnable task) {
 return e.submit(task);
 }
 public <T> Future<T> submit(Callable<T> task) {
 return e.submit(task);
 }
 public <T> Future<T> submit(Runnable task, T result) {
 return e.submit(task, result);
 }
 public <T> List<Future<T>> invokeAll(Collection<? extends Callable<T>> tasks)
 throws InterruptedException {
 return e.invokeAll(tasks);
 }
 public <T> List<Future<T>> invokeAll(Collection<? extends Callable<T>> tasks,
 long timeout, TimeUnit unit)
 throws InterruptedException {
 return e.invokeAll(tasks, timeout, unit);
 }
 public <T> T invokeAny(Collection<? extends Callable<T>> tasks)
 throws InterruptedException, ExecutionException {
 return e.invokeAny(tasks);
 }
 public <T> T invokeAny(Collection<? extends Callable<T>> tasks,
 long timeout, TimeUnit unit)
 throws InterruptedException, ExecutionException, TimeoutException {
 return e.invokeAny(tasks, timeout, unit);
 }
 }
 
 static class FinalizableDelegatedExecutorService
 extends DelegatedExecutorService {
 FinalizableDelegatedExecutorService(ExecutorService executor) {
 super(executor);
 }
 protected void finalize() {
 super.shutdown();
 }
 }
 
 /**
 * A wrapper class that exposes only the ScheduledExecutorService
 * methods of a ScheduledExecutorService implementation.
 */
 static class DelegatedScheduledExecutorService
 extends DelegatedExecutorService
 implements ScheduledExecutorService {
 private final ScheduledExecutorService e;
 DelegatedScheduledExecutorService(ScheduledExecutorService executor) {
 super(executor);
 e = executor;
 }
 public ScheduledFuture<?> schedule(Runnable command, long delay, TimeUnit unit) {
 return e.schedule(command, delay, unit);
 }
 public <V> ScheduledFuture<V> schedule(Callable<V> callable, long delay, TimeUnit unit) {
 return e.schedule(callable, delay, unit);
 }
 public ScheduledFuture<?> scheduleAtFixedRate(Runnable command, long initialDelay, long period, TimeUnit unit) {
 return e.scheduleAtFixedRate(command, initialDelay, period, unit);
 }
 public ScheduledFuture<?> scheduleWithFixedDelay(Runnable command, long initialDelay, long delay, TimeUnit unit) {
 return e.scheduleWithFixedDelay(command, initialDelay, delay, unit);
 }
 }
 
 /** Cannot instantiate. */
 private Executors() {}
 }

# 队列与线程池结合 #

以阿里云OSS文件存储为例：

package com.forestar.aliyun.service.oss.queue;
 
 import com.forestar.aliyun.service.oss.bean.FileInfo;
 import com.forestar.aliyun.service.oss.tds.OssHttpService;
 import lombok.extern.slf4j.Slf4j;
 import java.util.concurrent.ConcurrentLinkedQueue;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.concurrent.atomic.AtomicLong;
 
 /**
 * @Copyright: 2019-2021
 * @FileName: FileUploadQueue.java
 * @Author: PJL
 * @Date: 2020/12/3 10:35
 * @Description: 文件上传队列
 */
 @Slf4j
 public class FileUploadQueue {
 
 private ConcurrentLinkedQueue<FileInfo> queue = new ConcurrentLinkedQueue<>();
 
 private AtomicLong total = new AtomicLong(0);
 
 private AtomicLong completed = new AtomicLong(0);
 
 private AtomicInteger exeCount = new AtomicInteger(0);
 
 private ExecutorService executorService;
 
 private OssHttpService ossHttpService;
 
 private Boolean started = false;
 
 private int poolSize;
 
 /**
 * 文件上传队列初始化
 *
 * @param ossHttpService
 * @param executorPoolSize
 * @param startConsumer
 */
 public FileUploadQueue(OssHttpService ossHttpService, int executorPoolSize, boolean startConsumer) {
 this.poolSize = executorPoolSize;
 this.ossHttpService = ossHttpService;
 this.executorService = Executors.newWorkStealingPool(this.poolSize);
 if (startConsumer) {
 this.start();
 }
 }
 
 /**
 * 文件入队列
 *
 * @param fileInfo
 */
 public Long enqueue(FileInfo fileInfo) {
 queue.add(fileInfo);
 return total.incrementAndGet();
 }
 
 /**
 * 空闲判断
 *
 * @return
 */
 public Boolean isUnFull() {
 return exeCount.get() < poolSize;
 }
 
 /**
 * 空闲判断
 *
 * @return
 */
 public Boolean isBusy() {
 return exeCount.get() > 5 && isUnFull();
 }
 
 /**
 * 空闲判断
 *
 * @return
 */
 public Boolean isIdle() {
 return exeCount.get() <= 5;
 }
 
 /**
 * 开启消费线程
 */
 public void start() {
 if (!started) {
 new Thread(() -> {
 while (true) {
 try {
 int count = queue.size();
 // 线程池消费
 if (count > 0 && isUnFull()) {
 consumer();
 }
 // 忙碌延长休眠
 if (count > 0 && isBusy()) {
 Thread.sleep(100);
 }
 // 空闲缩短休眠
 else if (count > 0 && isIdle()) {
 Thread.sleep(50);
 }else{
 Thread.sleep(1000);
 }
 } catch (InterruptedException e) {
 e.printStackTrace();
 }
 }
 }).start();
 started = true;
 }
 }
 
 /**
 * 执行业务处理
 */
 private void consumer() {
 FileInfo fileInfo = queue.poll();
 if (null != fileInfo) {
 // 增加线程占用数量
 exeCount.incrementAndGet();
 // 提交执行任务
 executorService.submit(() -> {
 // 处理业务数据
 ossHttpService.syncToAliyunOss(fileInfo);
 // 执行个数增加
 completed.incrementAndGet();
 // 线程池占用减少
 if (exeCount.get() > 0) {
 exeCount.decrementAndGet();
 }
 // 打印处理进度
 log.info("===队列消费进度==={}/{}", completed.get(), total.get());
 
 // 处理完成通知
 if (completed.get() == total.get()) {
 log.info("=====================所有文件上传完成!=======================");
 }
 });
 }
 }
 
 }

最终我们得到类似下面的效果输出：

2020-12-03 12:38:08.802  INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService     : >>>开始解析....D:/TDSpath/list.txt
    2020-12-03 12:38:08.804  INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService     : httpUrl = http://remote_host/upload/eventAttach/eventpic/original/202007/20200711/05290729-bf44-41c6-8da7-44251e131d15.jpg , objectName = /upload/eventAttach/eventpic/original/202007/20200711/05290729-bf44-41c6-8da7-44251e131d15.jpg
    2020-12-03 12:38:08.804  INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService     : ==文件队列长度== size =1
    2020-12-03 12:38:08.805  INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService     : httpUrl = http://remote_host/upload/eventAttach/eventpic/thumb/202007/20200711/05290729-bf44-41c6-8da7-44251e131d15.jpg , objectName = /upload/eventAttach/eventpic/thumb/202007/20200711/05290729-bf44-41c6-8da7-44251e131d15.jpg
    2020-12-03 12:38:08.805  INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService     : ==文件队列长度== size =2
    2020-12-03 12:38:08.805  INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService     : httpUrl = http://remote_host/upload/eventAttach/eventpic/original/202007/20200711/15d1724a-a969-41f7-abe8-171904a746da.jpg , objectName = /upload/eventAttach/eventpic/original/202007/20200711/15d1724a-a969-41f7-abe8-171904a746da.jpg
    2020-12-03 12:38:08.805  INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService     : ==文件队列长度== size =3
    2020-12-03 12:38:08.806  INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService     : httpUrl = http://remote_host/upload/eventAttach/eventpic/thumb/202007/20200711/15d1724a-a969-41f7-abe8-171904a746da.jpg , objectName = /upload/eventAttach/eventpic/thumb/202007/20200711/15d1724a-a969-41f7-abe8-171904a746da.jpg
    2020-12-03 12:38:08.807  INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService     : ==文件队列长度== size =4
    2020-12-03 12:38:09.805  INFO 9792 --- [Pool-3-worker-9] c.f.a.service.oss.queue.FileUploadQueue  : ===队列消费进度===1/4
    2020-12-03 12:38:09.856  INFO 9792 --- [Pool-3-worker-9] c.f.a.service.oss.queue.FileUploadQueue  : ===队列消费进度===2/4
    2020-12-03 12:38:09.907  INFO 9792 --- [Pool-3-worker-9] c.f.a.service.oss.queue.FileUploadQueue  : ===队列消费进度===3/4
    2020-12-03 12:38:09.957  INFO 9792 --- [Pool-3-worker-9] c.f.a.service.oss.queue.FileUploadQueue  : ===队列消费进度===4/4
    2020-12-03 12:38:09.957  INFO 9792 --- [Pool-3-worker-9] c.f.a.service.oss.queue.FileUploadQueue  : =====================所有文件上传完成!=======================

[70]: /images/20221120/ae8d6af4e0e0435dbaaae0fa7a441c8f.png